Come meet us at CHI-NOG 10!
Come meet us at CHI-NOG 10!
TELNET Signals, our experience with Low-End Hosting Providers and other updates
As we progress the mission of Dataplane.org, it brings us a great sense of accomplishment to have our first official speaking engagement at Chicago Network Operators Group (CHI-NOG) 10. The event is on October 10th 2022 at the West Wolf Point Plaza in Chicago. Our very own Bill Eaheart will present “How NOT to Get Rich Quick: Building an Infrastructure Measurement NFP”. The full CHI-NOG 10 agenda can be found here. Dataplane.org’s co-founders will all be present and have the coveted Dataplane.org stickers available.
TELNET signals
We know lots of people, organizations, and systems are fetching our signal data on a regular basis. However, we only very occasionally receive feedback about what we’re doing. We consider no news to be good news, but when we receive praise it is the best news. We recently heard from someone who used our TELNET signal data for a summer project. This individual stated that our signal data “revealed something quite uncommon but ‘admirable’” and went on to say how our data contains entries “not found in ANY” (emphasis theirs) other well known sources of data. Furthermore, they found no false positives. This feedback is immensely gratifying to receive. We thank this individual for conveying it to us. This motivates us to keep doing more of what we’re doing.
As a small recognition of the importance our TELNET signal data has had for at least some in the Internet community, we took a quick look at frequency usage of TCP source ports for all these events in August. The plot below shows how much more frequent the high numbered ports are seen than lower ones. This perhaps is not too surprising as it seems to reflect modern OS ephemeral source port selection strategies, but it is interesting to see clear steps in the distribution.
Experiences with Low-End Hosting Providers
Our distributed sensor network is well suited to run on minimal hardware resources and rented from suppliers referred to as “low-end” hosting providers. The low-end market is personified by small, sometimes short-lived companies that can provide sufficient, but perhaps less predictable service guarantees than the well-known hosting behemoths. Rather than building out our own dedicated, global, costly network and server infrastructure or relying on volunteers to host systems for us, our low-end provider strategy has reached a reasonable level of predictability and consistency with minimal operational cost. Choosing and working with low-end providers however can be a bit of an art so we thought we’d share some of our experiences after many years of venturing down this path.
First, what can we expect to get and what will it cost? Our low-end systems range from free (yes, no kidding) to $40 per month with many in between. Many of our systems have been obtained through special offers, especially during special sales events such as Black Friday. Today, the lowest of the low virtual machines (VMs), also known as virtual private server (VPS), on average will have 1GB of RAM, 10GB of disk space, 1 virtual CPU, and an allowance of your “fair share” of a 1 Gb/s LAN housing dozens of other VMs. A premium is paid for exotic locations where power, IP addressing, space, and connectivity are less abundant. Many of the owners and operators of low-end hosting tend to be young hobbyists who have purchased one or more dedicated servers and some IP addresses from larger providers. More established low-end providers tend to own their own hardware, colocation space, and an autonomous system number (ASN). Some low-end providers also have their own provider independent IP address blocks, but a surprising number of them do not and this brings up one easy-to-overlook operational challenge.
Many low-end providers are associated with IP addresses from their upstream providers through a process like ARIN’s Shared WHOIS Project (SWIP) or with an inetnum object from the LIR at RIPE. This is all well and good until the low-end host decides to change upstreams or, as is sometimes the case, an upstream decides to sever ties with the low-end host. IP address renumbering is a fairly common occurrence in the low-end world. We’ve seen address changes as many as three or four times over the course of a few years. This is a relatively minor inconvenience, but we still will have to update documentation, reconfigure our remote management host inventory database, change DNS mappings that may be used, refresh public keys that may be tied to an IP address (e.g., SSH server keys), and so on. Furthermore, since many low-end providers not only lease address space, but also host hardware, the physical location of VMs may change. This means we may need to recalibrate our measurements and monitoring when that property factors into the use of our data.
Over the course of our existence we’ve used well over 100 hosting providers. Many have come and gone, sometimes due to their demise as an operation, other times as a result of our decision to take our business elsewhere. When we have decided to leave a provider, it is usually a combination of two factors that lead us to conclude the service is of insufficient value. One is that the provider’s service has been too inconsistent. The other is that the service lacks a unique property (e.g., exotic location) that would justify the inconsistency.
Despite the risks, there are a number of properties that attract us to certain low-end providers. The first is cost, but cost will only get us to look at the provider. It is rarely enough alone. We’ve seen incredibly attractive pricing appear from time-to-time, but we would rarely enter into a relationship with a provider that only came into existence in the last few months for example.
Our hosting costs would probably be at least double, maybe triple what they are now if we used only the large well established hosting providers. Our frugality has kept costs down with relatively minimal operational burden. We’ve learned to identify low-end providers that are likely to fulfill their promises, while avoiding those that would be more trouble than they are worth. Thankfully, at least in the case of a sensor system, no one host is critical. Individual host instability or downtime is largely inconsequential to the overall measurement and monitoring we do. This allows us to use providers that would be unacceptable in many business scenarios. It also means we can fix problems at a relatively leisurely pace, which reduces our overall stress and prevents us from having to work long or off hours. As an added bonus, we don’t bother our providers very often. If any one system, or even an entire provider’s network is down for a day or two, we have enough diversity that limits the overall impact. Providers probably like having us as customers as well. Most of the providers we work with we would not recommend for general purpose use, but you can check out our current list of commercial hosting providers we are currently partnered with at our server hosting repo.
Updates on Dataplane.org’s website
We’ve been making minor updates and tweaks to our website at https://dataplane.org to ensure mobile friendliness as well as functional parity across browsers which may, or may not, use javascript and other advanced features. While there are only a few of us touching the website, we’ve found that maintaining a consistency across pages through just basic text editing of html and css can be a cumbersome task. The latest version that you are viewing has been created utilizing the static site generator Hugo.
Hugo has allowed us to separate the content of our website from the design, allowing us to easily update content and design separately and uniformly with a simple workflow. This separation has allowed us to make changes at a faster pace and gives us the ability to deploy changes more rapidly as we automate our build pipeline from our git repositories. Our content is solely in a markdown language which provides a quick way to create new content, or update existing content.
As far as technical design principles go, we have found that a mobile first approach with industry accepted readability requirements is the best approach for a clean and easily read site. We found the base of the Skeleton CSS framework provides a clean, modern look and feel on both mobile and desktop browsers. To ensure usability without javascript, and other advanced features, we’ve tried to limit the usage of javascript to our statistics pages which use Chart.js. Other interactions such as drop downs and svg fallback to pngs have been done using standard css selectors, css animations and html tags.
Updates on Operational concerns of running a NFP
As techies, formalizing accounting and banking services is not at the top of our interest list. Still, we clearly recognize that fiscal responsibility is critical to operating a sound and compliant nonprofit organization.
We are shifting operations payment functionality away from personal banking accounts to business checking accounts, which is quite the undertaking with over 50 vendors who provide services to Dataplane.org. It was relatively easy to avoid monthly fees for opening a business checking account, but a slight surprise was the range of monthly transaction charges. The differentiators we’re considering are fee structures, accessibility to online tools, and the existence of a physical branch. We anticipate one way to avoid monthly transaction fees will be to utilize PayPal to execute charges via credit card instead of initiating withdrawals against the business checking account.
As a growing nonprofit business, we focus on managing our finances properly by reviewing options for tracking accounts payable and account receivables (well, for now, all we have are accounts payable), understanding our monthly cash outputs, and preparing for tax season with some much-appreciated tax write-offs. Do we manage finances through files, databases, and spreadsheets? Do we look to cloud options for accounting software? What functions do we need now, and how will we scale up with expansion? These are a few of the many questions we are addressing to understand the capabilities we require and the range of price tags. Selection process considerations include cost, ease of use, features, integrations, and scalability.
We welcome feedback on any items covered in our update or suggestions for improvement.
Feel free to reach out via email, Twitter, or Slack (request an invite if you need one).