Stop calling everything AI

Stop calling everything AI

Self Hosted vs. NBN Co

🌐 Self Hosted vs. NBN Co

TL;DR: How I (James) and my neighbour (Mike) fought NBN Co for ~10 years to get Fibre to the Premises rolled out in our local area. Many said “good luck”, and “NBN will never budge”, etc, but we stuck to our guns and often responded “just watch us”. Here we are a decade later, finally with Fibre to the Premises. Fibre is sooo great. 😊 We now experience a guaranteed reliability of 99.95% SLA instead of the daily drop-outs and barely making one 9’s of reliability, ~2-3ms latency down from ~10ms and increased bandwidth! In the end it was all thanks to one wonderful local politician, Elizabeth WatsonBrown (Greens Federal MP for Ryan)

Background 🗞️

In the land down under, where the sun kisses the blue waters and the vast outback stretches endlessly, the Australian government embarked on an ambitious project in 2009. NBN Co was born with the noble mission of connecting 90% of Australian households and businesses to a high-speed national broadband network by 2021. The initial plan, introduced by Kevin Rudd’s Labor government, called for the implementation of Fibre to the Premise (FTTP) at an estimated cost of A$37.4 billion.

The first taste of this digital revolution came to the isolated island of Tasmania around July 2010. The island’s businesses and educational institutions particularly stood to gain from this upgrade, with improved connectivity set to open new opportunities for growth and innovation. However, not every Tasmanian resident was as quick to embrace the new technology. Only 10% of the island’s population in the initial target areas chose to sign up for the NBN service at that time.

In 2010 Tony Abbott, the leader of the opposition Liberal Party, declared in parliament:

We will demolish NBN when we come into power!

Undeterred by this initial low adoption rate, and political turmoil, the NBN rollout continued onto the mainland, with the first customers in Armidale being connected around April 2011. The NBN Co team remained optimistic that more Tasmanians, as well as Australians across the country, would come to recognise the value of this revolutionary new infrastructure and join the digital revolution in due course.

In September 2013, the Liberal Party, under the leadership of Tony Abbott, won the Federal election. Malcolm Turnbull, the Minister for Communications, unveiled a revised NBN plan that aimed to speed up the completion of the rollout to 2019 and lower the overall cost to A$29.45 billion. This new plan involved a “multi-tech mix” of FTTN (Fibre to the Node) and FTTC (Fibre to the Curb).

What is FTTN, FTTC and FTTP anyway? ⁉️

FTTN (Fibre to the Node) involves bringing fibre-optic cable to a node in the broad area, whilst FTTC (Fibre to the Curb) brings fibre to the curb outside the premises to be connected, much closer than a node.

In both cases, FTTN and FTTC involve the use of VDSL (very-high bitrate digital subscriber line) technology over copper lines (often decades old and degraded) for the final “last mile” connection to each premise. This technology was already considered outdated and notoriously unreliable.

FTTP (Fibre to the Premises), however, is the most advanced and desirable technology, bringing fibre-optic cable all the way from the exchange to each individual premise. This technology provides faster speeds, greater reliability, and better overall performance than FTTN or FTTC.

The political madness 🤦‍♂️

Throughout this journey, we explored other avenues such as a community network, alternative providers and even the cost of paying NBN Co to upgrade the technology in the area. At one point, we went through the process of getting a “build quote” from NBN Co to upgrade the technology, called the “Technology Choice Program”. After going through the fine print we realised that the quoted A$13k per premise was in fact just an estimate and the price could go up and it would take many months before NBN would even start the work! Amazingly some Australians actually went through this and were charged up to A$40k each.

We also tried to get our local Federal MP to encourage NBN to include our area in their fibre roll-out plan. In June 2016 we met Jane Prentice (our Liberal Federal MP for Ryan) at her office who said that the NBN roll-out plan had been designed by the previous Labor government to prioritise Labor safe seats and marginal constituencies. Many safe Liberal seats like Ryan were not included in the plan’s initial stages. She said that the contract with NBN’s roll-out implementers included a financial penalty if the plan was changed. Therefore, she said that it was too expensive to change the “Labor” roll-out plan sequence. When we did finally get connected to the NBN network it was the Liberals' FTTN.

In the run up to the September 2019 Federal election our local Liberal Party branch removed Jane Prentice as their candidate and replaced her with Julian Simmonds who went on to win the seat for the Liberals. In June 2021 we met with Julian Simmonds at his office. We pointed out to him that many suburbs in Ryan had the unreliable FTTN connection to the NBN and that our suburb was still not on the NBN’s updated roll-out plan for upgrading to FTTP. He promised to check with NBN why this was the case, and when our suburb would be upgraded. After a few months his office contacted us to say that in fact we were connected to the NBN (by FTTN) and so there is no need for him to do anything further.

In the 2022 Federal election Julian Simmonds was defeated by the Greens candidate Elizabeth Watson-Brown. We went to meet Elizabeth Watson-Brown on 24 August 2022 and explained to her our problem with the NBN service in our area. She told us that she had heard similar complaints from a number of her constituents and undertook to help get us on to the NBN FTTP roll-out plan. Two weeks later she made a major speech in parliament specifically on the many failures of NBN and the embarrassing state of affairs we found ourselves (Australia) in, as “one of the wealthiest countries in the world per capita”. She also spoke about the specific problems of a number of suburbs in Ryan. As a result of this speech and her other contacts directly with NBN, our suburb was finally included in the NBN FTTP roll-out plan in early 2023.

This is her speech in parliament:

She subsequently wrote a letter to the Minister for Communications in November 2023 requesting that various suburbs in Ryan be added to the NBN FTTP roll-out. After receiving no reply from her, Elizabeth’s team organised a letterbox drop to all premises in Ryan that did not have FTTP – we volunteered for this delivery in our suburb. In this letterbox drop residents were invited to write to the Minister about how unhappy they were with their current NBN connections. This was followed-up with a second letter to the Minister from Elizabeth in early 2023.

As a result our suburb, and other Ryan suburbs, was finally added to the NBN FTTP roll-out plan in mid-2023.

The Horrors of Copper 😱

Before getting FTTP, we were stuck with copper lines for our internet connection. Copper lines are prone to various issues such as interference, electrical storms, and degradation over long distances. This results in slower speeds, dropped connections, and poor overall performance.

In our street, we had about ~400-500 m of copper between our premises and the “node”. The node runs the VDSL equipment called a DSLAM where it and your modem establish high-frequency signals. Others in the wider area had to deal with much greater distances, sometimes as much as 1 km or more. The copper from the old PSTN (Plain Simple Telephone Network) days in our area was built around ~20- 30 years ago and laid in pits filled with asbestos and rotting in the ground. Here’s a picture of our pit and the state it was in:

Furthermore, copper lines cannot support the high-speed demands of modern technology, making it difficult to stream video content or work from home effectively. We found ourselves constantly frustrated with our slow and unreliable internet connection, which made working and streaming a challenge.

If you were ~400-500 m from the “node” you could sustain at most ~50-60 Mbps downstream and ~20-30 Mbps upstream. With typical households, working from home and multiple video streams and conferences it was challenging and frustrating to get work done. On top of which you would be lucky if you could hold an active connection for more than a day before dropping out for up to 5 minutes, sometimes multiple times a day at the most inconvenient times causing immense frustration and lost time.

If you were further from the “node”, things were much much worse. Here’s a screenshot of some of the worst times for reliability we had to suffer through:

Sadly many residents in our area we spoke to, some of which also worked remotely, found all this to be “normal” and would often blame their “laptop”, not realising how bad the networking infrastructure really was!

Our FTTP Roll-Out 🚧

As you have seen, after years of persistence, and lobbying in 2022/3 by Elizabeth Watson-Brown (our local Federal Greens MP), we finally managed to secure FTTP for our area.

James’s FTTP installation was being case managed by NBN Co at this stage, as he had tirelessly monitored the quality and reliability for years and spent countless hours filing faults and arguing with NBN Co about the state of their infrastructure and how poor it was. Mike had also sporadically filed service complaints to his ISP Telstra over the years, most of which resulted in their technicians reporting that the problems were largely the result of our copper connections to the node. Most of these complaints were not reported to NBN by Telstra.

The roll-out process in our area took about 8 months once it was started in mid2023 by NBN and its sub-contractors. After the many different crews had finished rolling fibre down through the major roads and streets, cleaning and replacing old pits and expanding the fibre network to the pits in our suburb. The final fibre connection from the pits serving the individual customers to their premises had to be requested by the customer’s own ISP. Once this request had been received by NBN, arrangements were made for the installation of the fibre connecting their premises to the pit serving them. As neighbours, James and Mike share the same pit which is in the footpath outside their houses.

After James put his order in with his ISP (Aussie Broadband), an NBN technician came out to assess what the installation of the fibre from the pit would entail. At first it seemed that there would be further delays as the technician wasn’t able to locate the conduit and it would have been impossible to bring the fibre to where it needed to go in his house. Thankfully the NBN technician, after some brain-storming, came up with a plan that worked, so he proceeded to install the NTD (Network Termination Device) that goes inside the premises and the utility box that goes on the outside (about 1 to 1.5 m from the NTD). Then he placed an order for the “civil crew” from NBN to come back to trench and install new conduit and splice in the fibre from the pit to the utility box.

By this stage James was quite anxious to start enjoying the new FTTP technology, so kept on top of progress and contacted Aussie Broadband every 2 days to see where things were at. To his amazement and horror, NBN had booked the “civil crew” about 2 months out from the initial internal installation of the NTD and utility box. As this was quite unacceptable, even for the poor Aussie Broadband customer service rep on the other side of the phone call, a request was put through to have this date brought much closer.

A few days later on a Saturday, a phone call from the “civil crew” team lead said they’d come up and complete the job that day, on a weekend! Apparently some of these contractors were happy to work weekends. After half a day of trenching and digging and laying down conduit in the rain, the new fibre was installed from the pit to the utility box.

Surprisingly these “civil crews” are unable to splice and connect the fibre in the utility box to the fibre in the NTD as they are often unlicensed and don’t have the tools. The two fibre cables are of different thickness, with the thicker one running in the conduit from the pit to the utility box and the thinner one pre-installed within the NTD(s) by the manufacturer.

Two weeks later, the same NBN technician was back to complete the last 15 minutes worth of work, to splice and connect the external fibre from the pit (now in the utility box) to the smaller/thinner fibre in the NTD.

An hour or so later, James was up and running on 250 Mbps downstream, 100 Mbps upstream with ~2-3 ms latency (previously it had been 10 ms!). The difference in performance is night and day. He is now enjoying lightning-fast speeds, reliable connectivity, and the ability to use modern technology without any hassle.

Before FTTP, James' internet connection had high latency, around 10 ms, which made online activities such as gaming or video conferencing less enjoyable. Additionally, the connection was unreliable, with an SLA (Service Level Agreement) of barely making one “9” and experiencing daily drop-outs.

After getting FTTP, James now enjoys significantly lower latency, around 2-3 ms, which has noticeably improved our online experience in terms of responsiveness and interactivity. Furthermore, the guaranteed reliability of the connection is much higher, with an SLA of 99.95%, ensuring a more stable and dependable internet connection for our daily needs.

Mike put in his request to his ISP (Telstra) a couple of weeks later than James did, but his experience was similar. The NBN technician who came to install his fibre connection from the pit to his premises arrived in mid-January. To celebrate this, Mike invited Elizabeth and one of her team members to join him and James for tea while Mike’s installation was taking place. Disappointingly, the NBN technician found that, similar to James' experience, Mike’s existing conduit running from the pit to his house was blocked. This meant that a “civil team” would also need to come and install new underground conduit before the final connection could be completed. The technician was thus only able to install Mike’s NTD and utility box during Elizabeth’s visit. As was the case for James, Mike was given a date for the civil works in 2 months' time. Fortunately, Mike was able to convince Telstra to request NBN to shorten the delay and so his final connection is scheduled for late February.

vDSL2 sucks NBN sucks Copper sucks

It continues to amaze me how NBN continues to operate. With over $50B AUD of taxpayer funds later (See NBN Project costs) folks like me that live in the suburbs continue to have less than ideal quality Internet service.

As of this post, I’m sitting on a vDSL2+ connection, with a Fibre to the Node backhaul, delivered over ~450m of Copper cable (last mile). At the time of writing, two LQD (Line Quality Diagnostics) tests later show the SNR/Noise Margin to be averaging around ~5db. 😳

If you do a simple Google search for “vDSL2 SNR” you will very quickly find:

The transmission noise margin limit is 6 dB, a value below which a VDSL connection is not guaranteed at all. If the noise margin is below 6 dB, frequent interruptions may occur. If the noise margin is greater than 10 dB, the line has good parameters for data transmission.25 Apr 2022

Anyone with experience in wireless technology, radio of electromagnetic radiation knows that a signal with such a low SNR is pretty piss poor. It’s amazing the service works at all (oh wait it doesn’t really, with 16 drop-outs in 48hrs 😱)

What can I do about this? 🤔 Nothing, absolutely nothing! 🤬 All I can do is continue to complain to my ISP, make posts like this, and continue to put political pressure on NBN Co who is run by folks (Stephen Rue as CEO), that have some pretty backwards ideas of economy and networking.

If anyone managed to reach this far, I’ve been battling with NBN Co for years now. I will continue to do battle with them on many fronts until I get some goddamn Fibre rolled down the street. The only thing I worry about the most, is that by the time that happens, I’ll have well and truly retired 🤣

#nbn #internet #vdsl #snr

Why is loading the timeline on the micro.Blog app Soo slow? 🤔

Why is loading the timeline on the micro.Blog app Soo slow? 🤔

Hey @manton Are posts from Activity Pub actors I follow suppose to show up like this with the acct: prefix?

Hey @manton Are posts from Activity Pub actors I follow suppose to show up like this with the acct: prefix?

Hey @manton wondering if you can help debug on your side for me why micro.blog isn’t looking up successfully, get a spinning search...

Hey @manton wondering if you can help debug on your side for me why micro.blog isn’t looking up successfully, get a spinning search wheel and no results, my activitypub user @james@yarn.mills.io (custom implemtnation I’m building). Works with a GTS (GotoSocial) instance so far, but not micro.blob :/

Please vote no 👎 for the Telemetry in the Go toolchain proposal 🙏 #golang #telemtry #justsayno

Please vote no 👎 for the Telemetry in the Go toolchain proposal 🙏 #golang #telemtry #justsayno

A “distributed network” does not make your app “decentralised” 🤦‍♂️

A “distributed network” does not make your app “decentralised” 🤦‍♂️

Now that Elon Musk owns Twitter™, it might be worthwhile thinking about other alternative social media ecosystems like Yarn.social based...

Now that Elon Musk owns Twitter™, it might be worthwhile thinking about other alternative social media ecosystems like Yarn.social based on the Twtxt format.

Protecting Internal Web Resources

TL;DR: This blog post is a write-up of the process I went through to setup a set of internal web resources and apps for a small company I am running in my spare time (providing a Single-Sign-On / SSO experience for internal users with web applications protected by flexible access policies including single and multi-factor authentication / two-factor authentication or 2FA).

Background

As I mentioned in the TL;DR above, I run a small software/technology company whereby I needed a way to stand up a few internal resources and web applications for our growing needs.

These needed to be protected and secured from unauthorized access. In addition to this, since the company’s primary mission is all about “self hosting” software and services, it also had to be self hosted or self-hostable.

There were numerous other options I explored but dismissed for various reasons including:

• Setting up an internal VPN
• Just exposing the web applications / resources directly without extra security.
• Using Basic or Digest Auth and a Reverse Proxy
• Using oauth2-proxy and some other solution based on oauth2-proxy that got spun into a commercial company.

Suffice to say a lot of existing solutions were either “too complicated” to setup, proprietary or commercial in some way or just didn’t quite fit the bill.

In the end I picked:

• Authelia - The Single Sign-On Multi-Factor portal for web apps

Why I picked Authelia?

I had my eye on Authelia for a while now, and today (based on needs and internal demands) I finally decided to take a serious look at it.

Immediately I noticed the Docker local bundle which then lead me to an example docker-compose setup. This is important to me as I’m quite a lazy sysadmin and I really hate being one, its boring and installing packages and writes tonnes of configuration is just well not enjoyable.

Do not worry my Kubernetes friends, Deployments has you covered with a k8s deployment (because k8s is an operating system rigt?! 🙄), there’s even a Bare-Metal installation, and I’m quite sure you could adapt Authelia in any environment.

The other reasons I picked Authelia are:

• It provides a simple built-in Authentication layer using a simple YAML file for defining Users, Credentials and other metadata like Display Name, Groups, Email Address, etc.
• It provides a really flexible way to define Access Control so it was very easy for me to setup all the rules I needed for public access, internal access and everything in-between.

You guys/gals over at micro.blog see any new users join over Twitter™ buyout debacle from Elon Musk? 🤔We haven’t seen any join...

You guys/gals over at micro.blog see any new users join over Twitter™ buyout debacle from Elon Musk? 🤔We haven’t seen any join Yarn.social (yet) but allegedly ~30k joined Mastodon?

yarnd v0.13 - Aluminium Amarok

Today we announced release v0.13.0 of the Yarn.social backend yarnd that now powers a network of 15 pods around the globe.

You can find the release here:

• https://git.mills.io/yarnsocial/yarn/releases/tag/0.13.0

Yarn/Twtxt (Yarn.social is based on Twtxt) continues to grow steadily every day, and every month or so we see a new independent Pod (what we call instances) come online! 🥳

We are now entering an exciting phase of the project where we will “freeze” the project for a while while we focus on improvements to documentation, relicensing, improved builtin pages, entering the Vultr marketplace and deployment and operator guides.

If you’ve ever wished there was an alternative to Twitter™ (maybe you’re tired of giving up your privacy?) or Mastodon™ (maybe you’ve tried and found it too hard?) then come check out Yarn.social 🤗

Future of Yarn.social

Today I’m going to talk about Yarn.social’s future, a roadmap into where we’re going and thinking. I’ll also write a little about it’s history of where we came from and highlight how Yarn.social is different and in my opinion better.

What?

For those of you new to my blog or Yarn.social; Yarn.social is a decentralised social media platform, a microBlogging platform. It was originally created in July of 2020 and was then called Twt.social – since then it has going through a rebranding and seen huge improvements in the user experience and features that make up the platform.

If you are somewhat familiar or used to Twtter™, Yarn.social has a similar feel, but the key things to note are:

• It is 100% decentralised, there is no single server, entity or organisation in “control”
• There is no metadata collection of any kind or tracking of users
• There are absolutely no advertising

Where?

Today Yarn.social consists of:

• An ever improving backend called yarnd that powers every single and multi user pod (what we call instances that make up the network)
• An ever improving default theme providing a decent user experience for all new pod owner/operators and users
• A new but basic landing page at Yarn.social that has most of the content we need in place
• An updated native Mobile App actively being worked on but that is still in internal development and alpha testing
• A global search engine and crawler that whilst functional still needs some work
• A feed service that supports subscribing to Twitter and RSS/Atom feeds
• 4x multi-user pods and 7 single-user pods
• 415 feeds across the network and general ecosystem which have produced and consumed some whopping ~256,000 Yarns!

When?

Tomorrow it is my hope that we:

• Continue to improve the general user experience of the default web interface’s theme
• Complete the re-development of the native Mobile App and get it published to the App and Play stores
• Redesign the landing page at Yarn.social with a better layout and look

Not literally tomorrow of course 😂 but this is more of an ongoing set of efforts I’d like to see continue as well as some immediate long-over due goals.

In addition to the above, I would really like to get a Not-for-Profit (NFP) organisation setup and created for Yarn.social – Why? I believe Yarn.social and all the work we’ve put into the platform, it’s components, it’s specs and so forth should remain wholly in the hands of the community and in my opinion we should continue to govern ourselves this way. I never intended to create yet another privacy eroding social platform (you know the ones that profit off of user data and metadata and care nothing but user engagement on their centralised platform so drive up advertising revenue).

So over the holidays:

• Seek help in setting up a Not-for-Profit

Or:

• Sit down and work through the required paperwork and just “get it done”

Once this is done we can move on to other things that are in the works and eventual long-term goals such as:

• Getting Yarn.social yarnd into the Marketplace of Cloud providers such as Vultr and DigitalOcean – The only blocker right now is an official organisation and support structure (hence the need and desire to create a Not-for-Profit above).
• Seeking funding from either the community through donations and/or through philanthropy investors so that we can continue to support the development and maintenance of the platform over time.
• Redesigning and developing the Hosted Pods platform as a service to create a level of convenience for new users and new pod owner/operators that either cannot run/host their own pod due to lack of experience or technical adeptness or prefer to have someone else manage the “hosting” and “maintenance”.
• Building a new and more modern sleeker Web Application frontend ala Race-style (but without using any heavy frameworks like React!) – This is a bit controversial (or has been) for some, especially those that enjoy the lightweight ness of the server-side web interface that exists on all pods today. But it becomes difficult to build some types of user experiences this way and it is also hard to maintain a native Mobile App experience that uses a completely different set of technologies, codebase and even programming language! The idea here (and has been already started somewhat) is to build a Progressive Web App / Single Page App that utilises the backend API of yarnd and Mithril JS as well as some good CSS to create a rich, interactive, almost-native app-like experience. This app would be hosted centrally at either app.yarn.social or yarn.social/app and serve as a front to any pod, it may also be builtin to yarnd itself as an optional web interface.

And finally, but most importantly to me:

• Continue the discussion of building and supporting the idea of “Private Feeds” – We’ve talked about the use-cases for this over some months, and whilst this itself may be a lot of work, and there may be other ways to solve similar use-cases, I feel it is important to also build ways in which people are also able to “socialise” and “microBlob” privately.
• Reignite a much older discussion and idea of (what was originally called) “Supervised Pods”, but more recently I’ve thought of this as “Supervised Accounts”. This is the idea that (and something very important to me) that as a Father of two young daughters who will very likely want to get into such things as “social media”, that they should be able to do so BUT not on any of these other privacy eroding garbage that exists out there AND should be protected from predators, bullying, harassment and so forth (things that are in violation of all pod’s default abuse policy anyway and would result in a Pod Admin/Operator just nuking your account).

And so this concludes the general vision and goals of Yarn.social. Over time it is my great hope that the network continues to grow and continues to be cultivated. IT’s hard to put numbers on this because of the decentralised nature of Yarn.social / Twtxt in the first place; but let’s see if we can (over the next year) get to:

• 20+ Yarn.social pods
• 500+ Feeds/Users across the network and ecosystem
• 1M+ produced and consumed Yarns!

How?

Good question! This is a lot of work, it’s already been a lot of work. In only a few short months Yarn.social itself will reach it’s 1st official birthday (2nd if you count the fact it was once called Twt.social).

This can go either one of two ways in my view:

• Yarn.social continues to evolve and improve slowly over time with the help of contributors that donate what little spare time they have (including myself)

Amd/Or:

• I start to wind down my own day job over time so focus more on the things I love. I obviously can’t do this yet as I still need to also support my family.

And/Or:

• A nice philanthropist comes along and donates a sizeable chunk of money to the project because they see the value in what we’re building and then suddenly things go a lot faster! 😂

Why?

Many of the driving factors behind Yarn.social today that were originally talked about in a much older post Why an open, decentralised social media network like Twtxt and Twt.social matter are still true today.

I won’t restate in great detail what is already on the About Yarn.social page, but if I am to sum up everything into a single visionary statement it would be:

To create the best, unobtrusive and unfettered social medial platform to anyone that values their privacy and freedoms.

Hey @manton Would you be open to the idea of opening up micro.Blog to Yarn.social? 🤔 I understand that micro.Blog uses micro.pub as the...

Hey @manton Would you be open to the idea of opening up micro.Blog to Yarn.social? 🤔 I understand that micro.Blog uses micro.pub as the underlying protocol for it’s “micro blogging” right?

Hey @manton I got this email this morning from name.com – Just wondering if I could have my prologic.blog domain transferred to me?...

Hey @manton I got this email this morning from name.com – Just wondering if I could have my prologic.blog domain transferred to me? 🤔 Whilst I have enjoyed using my micro.blog service I’d like to move away off of micro.blog and host my own blog, which I intend to use my own static tool for.

Thank you for providing such a great service over the years I’ve used micro.blog 🙇‍♂️

I’ve deleted several of my other RSS/Atom feeds in my account, so users that follow me here on Micro.blog will no longer see content...

I’ve deleted several of my other RSS/Atom feeds in my account, so users that follow me here on Micro.blog will no longer see content from my Twtxt feed. Why? Well in truth whilst it’s a great feature honestly, I get next to zero interaction from anyone on Micro.blog.

Why I no longer trust Github

This is a blog post on why I no longer trust Github that Microsoft bought a few years ago (for a few billion) with my code. I will outlines the events that occurred, the thinking that lead me to the decisions that I made and my final thoughts.

Rest assured, I will continue to contribute to open source, write open source software, I just won’t host it on any corporate owned “Cloud” (SaaS) code hosting platform. I will self-host*.

A little about me

Those of you who don' know, my name is James Mills. I’ve been with Github since ~2008. I started contributing to open source and writing open source projects since ~2004 with my first project being a Python library that later turned into the circuits event-driven asynchronous library used by many (well before asyncio became a thing in Python many years later).

I’ve worked in numerous companies in Australia being a Backend Developer, Full Stack Engineer, DevOps Engineer, System Administrator, Network Administrator, Helpdesk Support and many other roles over the years. I’ve worked at Facebook for over 3 years as a Production Engineer where I earned much respect for my ability to “Get Stuff Done"™.

Over the course of some ~17 years of contributing to and writing open source projects I roughly estimate (really hard to actually measure) I’ve written between 500,000 to 1,000,000 lines of code.

Why did I leave Github?

I am primarily leaving Github because I no longer trust them as an entity to host my code. Code that I write and openly license under the MIT License. The recent announcement of Microsoft/Github’s Copilot which harvested source code from public Github repositories and used that as the training dataset for their machine learning models I feel is both a violation of the (©) Copyright I hold on source code I wrote but also a violation of the License agreement(s) I typically choose to license my works under.

Specifically the use of any of my works is subject to (because I use the same license normally):

Permissions

• Commercial use
• Modification
• Distribution
• Private use

Limitations

• Liability
• Warranty

Conditions

• License and copyright notice

It is the last one that is really important to me. You are free to use my works in any projects of your own, commercial or otherwise, as long as you attribute my work and maintain my License and Copyright. If you do not you are violation.

As a user of Copilot, if any works you derive (or in some cases is a direct copy/pasta) from my source code is in direction violation of the license of my work unless you:

• Know the source of the suggestions and snippets that came from Copilot
• AND acknowledge, attribute and maintain the License and Copyright of the original works.

Github’s own Terms of Service even states (if you bother to read it carefully):

This license does not grant GitHub the right to sell Your Content. It also does not grant GitHub the right to otherwise distribute or use Your Content outside of our provision of the Service, except that as part of the right to archive Your Content, GitHub may permit our partners to store and archive Your Content in public repositories in connection with the GitHub Arctic Code Vault and GitHub Archive Program.

Assuming Microsoft who wrote Copilot, a Visual Studio extension, intends to eventually sell this as a service to their users, they are in direct violation of Github’s own Terms of Service as stated above and captured here for posterity.

FWIW I used to up until now even pay Github for “extra features” and “privileges” on their platform. Things like unlimited private repositories, more actions, more storage, etc. No longer! I’ve now downgraded to the “Free” account.

So what happened?

Here follows a series of events, including screenshots for posterity (in case things get removed or re-worded, etc) of what happened over the last couple of weeks that eventually lead me to the decision to remove all of my source code from Github.

• Sun, Jul 4 3:34AM Slashdot posts about mixed reactions of Micorosft’s Copilot

• Sun, Jul 4 5:21AM A fellow open source developer talks about “Abandoning GitHub”

• Sun 4, 12:09PM I followup (_after reading about what’s ) with thoughts of my own and contemplating follow suit.

• Mon 5, 9:10PM Julia Reda posts about GitHub Copilot is not infringing your copyright.

  I don’t know who this Julia Reda is, but even though she’s right about Github/Microsoft not technically infringing on (©) Copyright per se, what has been (and is being done) is a direction violation of many open source licenses.

• Wed 7, 5:07PM I ask my social media and those that follow me their opinions.

• Thu 8, 6:18PM Github confirm that they didn’t just selectively use a subset of repositories with particular licenses, they scraped them all!

• Fri 9, 11:57PM Another open source developer announces leaving Github over the Copilot launch.

• 3:30PM I finally announce my decision to do the same.

So now what?

So now that I’ve migrated all of my projects and their source code off of Github to my own Self-Hosted platform, then deleted all content from Github (except my identity, which I will maintain); Now what?

Projects have moved

All my projects have a new home at git.mills.io. I’ve always had my own private Git Hosting platform (I used to use Gogs) but since this weekend I’ve decided to stand up Gitea instead as it has a few more features that make it a bit more convenient for collaborating with others and has a familiar feel and experience as compared with Github/Gitlab/etc.

I’ve also enabled Github OAuth as well so anyone that had previously contributed to my projects on Github, raised Issues or just want to continue to follow me work can do so with relative ease by simply signing in to https://git.mills.io with their Github identity.

Go libraries and import paths

Since I’ve been an avid Go developer for quite some years now and have written a considerable number of libraries and software (some quite popular), I feel you should be aware that import paths are now obviously broken and you need to update your own projects:

Change any import from:

import "github.com/prologic/<library>"

To:

import "git.mills.io/prologic/<library>"

At some point I will be standing up my own Go Vanity Package hosting to make the imports canonical, simpler and resistant to change no matter where the code lives.

Final thoughts

It’s been a crazy couple of years with the recent pandemic and all. Something fairly similar happened to a chat platform I had been a member of for over ~18 years, FreeNode IRC Network. It never ceases to amaze me what corporations can do to ruin a “good thing”.

I want to thank each and every one of you that have contribute to my own projects and accepted my own contributions via Github over the years. I hope that many of you will continue to follow my work and use many of my works as I continue to maintain and develop it under it’s new home (mine!).

Happy hacking! 🙇‍♂️

Considering deleting all source code from Github re Copilot vioations

II may also consider deleting all my source code repositories and forks from Github.

For more details see: twtxt.net/conv/xv2o…

Welcome to comment here also, I will respond here too!

# So I'm a Knucklehead eh?

So I’m a Knucklehead eh?

This small post is mostly in response to an article I saw this morning in my feeds: Building a search engine to rival Google could cost billions — and that’s not the only problem by James Purtill.

Challenge accepted!

I challenge anyone with a few spare tens of millions of $AUD to come talk to me about this. Put your money where your mouth is (so to speak). Does it really cost 100’s of Billions of dollars ro build, crawl and index the Web? I think not!

What, wait? Who are you?!

Who the hell am I you ask? Really?! You’re crazy!

Maybe 😉

I don’t expect most of you to know me too well (except those that do!), I’m just a software engineer that has worked in quite a few places over a period of more than 15 years, including several years at Facebook Inc. I’m very familiar with the kinds of Infrastructure and Engineering it takes to build “Web Scale” products like Search or a large-centralised Social Media!

It is quite sad to read other failed attempts at building a search engine, mostly Government backed attempts. The reality is a lot of government backed projects tend to be inefficient and the funds wasted before the project even has a chance to get off the ground or suffers from political issues (example: NBN Co in Australia where many of us still have Copper Last Mile over *DSL).

What would it take?

What would it cost to build an Aussie search engine? Well I put together one over a weekend called Spyda because my wife asked me to! I said I’d have it ready by Monday. It was a day late and launched on Tuesday.

Spyda is available at spyda.dev where you can currently search a measly 1,000+ pages. We’ve only been able to crawl and inex a thousand pages so far. This is becuase it runs on my home infrastructure which has an ordinary 100/40 Mbps VDSL NBN Co connection backing it (behind Cloudflare).

In other words it cost a day of my engineering time to put together, plus the cost of my home infrastructure (with initial outlay of $10k and $5k maintenance over the years).

So basically nothing so far to prove the concept is feasible. 😝

What now?

So let’s scale this up!

If someone were to approach me tomorrow offering to invest in my expertise through a new company founded to compete with the likes of Google, Facebook, Twitter and all these other tech-giants that have successfully built up a multi-Trillion dollar empire by profiting off the data collected from their “users”, come talk to me!

Let’s build an Aussie search engine, that’s Aussie built and owned, but does not sell its users data!

You can reach me at James dot Mills at shortcircuit dot net dot au – Or just Google™ me 🤣

When I add an RSS/Atom feed of mine here, am I the only one that sees its content?

When I add an RSS/Atom feed of mine here, am I the only one that sees its content?

Cross-posted to gist.github.com

Cross-posted to gist.github.com

Learn Go in ~5mins

Learn Go in ~5mins

This is inspired by A half-hour to learn Rust and Zig in 30 minutes.

Basics

Your first Go program as a classical “Hello World” is pretty simple:

First we create a workspace for our project:

$ mkdir hello

Next we create and initialize a Go module:

$ go mod init hello

Then we write some code using our favorite editor:

package main

import "fmt"

func main() {
  fmt.Println("Hello World!")
}

And finally we build and produce a binary:

$ go build

You should now have a hello binary in your workspace, if you run it you should also get the output:

$ ./hello
Hello World!

Variables

You can create variables in Go in one of two ways:

var x int

Other types include int, int32, int64, float32, float64, bool and string (and a few others…), there are also unsigned variants of the integer types prefixed with u, e.g: uint8 which is the same as a byte.

Or implicitly with inferred types by creating and assigning a value with:

x := 42

Values are assigned by using the = operator:

x = 1

Functions

Functions are declared with the func keyword:

func hello(name string) string {
  return fmt.Sprintf("Hello %s", name)
}

Functions with a return type must explicitly return a value.

Functions can return more than one value (commonly used to return errors and values):

func isEven(n int) (bool, error) {
  if n <= 0 {
    return false, fmt.Errorf("error: n must be > 0")
  }
  return n % 2 == 0, nil
}

Go also supports functions as first-class citizens and as such supports many aspects of functional programming, including closures, returning functions and passing functions around as values. For example:

func AddN(n int) func(x int) int {
	return func(x int) int {
		return x + n
	}
}

Structs

As Go is a multi-paradigm language, it also support “object orientated” programming by way of “structs” (borrowed from C). Objects / Structs are defined with the struct keyword:

type Account struct {
  Id:      int
  Balance: float64
}

Fields are defined similar to variables but with a colon : separating their name and type. Fields are accessed with the dot-operator .:

account := Account{}
fmt.Println("Balance: $%0.2f", account.Balance)

Methods

Structs (objects) can also have methods. Unlike other languages however Go does not support multiple-inheritance nor does it have classes (you can however embed structs into other structs).

Methods are created like functions but take a “receiver” as the first argument:

type Account struct {
  id  int
  bal float64
}

func (a *Account) String() string {
  return fmt.Sprintf("Account[%d]: $0.2f", a.id, a.bal)
}

func (a *Account) Dsposit(amt flaot64) float64 {
  a.bal += amt
  return a.bal
}

func (a *Account) Withdraw(amt float64) float64 {
  a.bal -= amt
  return a.bal
}

func (a *Account) Balance() float64 {
  return a.bal
}

These are called “pointer receiver” methods because the first argument is a pointer to a struct of type Account denoted by a *Account.

You can also define methods on a struct like this:

type Circle struct {
  Radius float64
}

func (c Circle) Area() float64 {
  return 3.14 * c.Radius * c.Radius
}

In this case methods cannot modify any part of the struct Circle, they can only read it’s fields. They are effectively “immutable”.

Arrays and Slices

Arrays are created with [T]like this:

var xs []int = []int{1, 2, 3, 4}

Arrays can also be created and appended to:

var xs []int
xs = append(xs, 1)
xs = append(xs, 2)
xs = append(xs, 3)

You can access an array’s elements by indexing:

xs[1]  // 2

You can also access a subset of an array by “slicing” it:

ys := xs[1:] // [2, 3]

You can iterate over an array/slice by using the range keyword:

for i, x := range xs {
  fmt.Printf("xs[%d] = %d\n", i, x)
}

Maps

Go has a builtin data structure for storing key/value pairs called maps (called hash table, hash map, dictionary or associative array in other languages).

You create a map by using the keyword map and defining a type for keys and type for values map[Tk]Tv, for example a map with keys as strings and values as integers can be defined as:

var counts map[string]int

You can assign values to a map just like arrays by using curly braces {...} where keys and values are separated by a colon :, for example:

var counts = map[string]int{
  "Apples": 4,
  "Oranges": 7,
}

Maps can be indexed by their keys just like arrays/slices:

counts["Apples"]  // 4

And iterated over similar to array/slices:

for key, value := range counts {
  fmt.Printf("%s: %d\n", key, value)
}

The only important thing to note about maps in Go is you must initialize a map before using it, a nil map will cause a program error and panic:

var counts map[string]int
counts["Apples"] = 7  // This will cause an error and panic!

You must initialize a map before use by using the make() function:

var counts map[string]int
counts = make(map[string]int)
counts["Apples"] = 7

Flow control structures

Go only has one looping construct as seen in the previous sections:

sum := 0
for i := 0; i < 10; i++ {
  sum += i
}

The basic for loop has three components separated by semicolons:

• the init statement: executed before the first iteration
• the condition expression: evaluated before every iteration
• the post statement: executed at the end of every iteration

If you omit the condition you effectively have an infinite loop:

for {
}
// This line is never reached!

Go has the usual if statement along with else if and else for branching:

var N = 42
func Guess(n int) string {
  if n == 42 {
    return "You got it!"
  } else if n < N {
    return "Too low! Try again..."
  } else {
    return "Too high! Try again..."
  }
}        

Note: The last else could have been omitted and been written as return "Too high~ Try again...", as it would have been functionally equivalent.

There is also a switch statement that can be used in place of multiple if and else if statements, for example:

func FizzBuzz(n int) string {
  switch n {
  case n % 15 == 0:
    return "FizzBuzz"
  case n % 3 == 0:
    return "Fizz"
  case n % 5 == 0:
    return "Buzz"
  default:
    return fmt.Sprintf("%d", n)
  }
}

Functions can be executed at the end of a function anywhere in your function by “deferring” their execution by using the defer keyword. This is commonly used to close resources automatically at the end of a function, for example:

package main

import (
  "os"
  "fmt"
)

func Goodbye(name string) {
  fmt.Printf("Goodbye %s", name)
}

func Hello(name string) {
  defer Goodbye(name)
  fmt.Printf("Hello %s", name)
}

func main() {
  user := os.Getenv("User")
  Hello(user)
}

This will output when run:

$ ./hello
Hello prologic
Goodbye prologic

Error handling

Errors are values in Go and you return them from functions. For example opening a file with os.Open returns a pointer to the open file and nil error on success, otherwise a nil pointer and the error that occurred:

f, err := os.Open("/path/to/file")

You check for errors like any other value:

f, err := os.Open("/path/to/file")
if err == nil {
  // do something with f
}

It is idiomatic Go to check for non-nil errors from functions and return early, for example:

func AppendFile(fn, text string) error {
  f, err := os.OpenFile(fn, os.O_CREATE|os.O_APPEND|os.WR_ONLY, 0644)
  if err != nil {
    return fmt.Errorf("error opening file for writing: %w", err)
  }
  defer f.Close()
  
  if _, err := f.Write([]byte(text)); err != nil {
    return fmt.Errorf("error writing text to fiel: %w", err)
  }
  
  return nil
}

Creating and import packages

Finally Go (like every other decent languages) has a module system where you can create packages and import them. We saw earlier In Basics how we create a module with go mod init when starting a new project.

Go packages are just a directory containing Go source code. The only difference is the top-line of each module (each *.go source file):

Create a Go package by first creating a directory for it:

$ mkdir shapes

And initializing it with go mod init:

$ cd shapes
$ go mod init github.com/prologic/shapes

Now let’s create a source module called circle.go using our favorite editor:

package shapes

type Circle struct {
  Radius float64
}

func (c Circle) String() string {
  return fmt.Sprintf("Circle(%0.2f)", c.Radius)
}

func (c Circle) Area() float64 {
  return 3.14 * c.Radius * c.Radius
}

It is important to note that in order to “export” functions, structs or package scoped variables or constants, they must be capitalized or the Go compiler will not export those symbols and you will not be able access them from importing the package.

Now create a Git repository on Github called “shapes” and push your package to it:

$ git init
$ git commit -a -m "Initial Commit"
$ git remote add origin git@github.com:prologic/shapes.git
$ git push -u origin master

You can import the new package shapes by using it’s fully qualified “importpath” as github.com/prologic/shapes. Go automatically knows hot to fetch and build the package given its import path.

Example:

Let’s create a simple program using the package github.com/prologic/shapes:

$ mkdir hello
$ go mod init hello

And let’s write the code for main.go using our favorite editor:

package main

import (
	"fmt"

	"github.com/prologic/shapes"
)

func main() {
	c := shapes.Circle{Radius: 5}
	fmt.Printf("Area of %s: %0.2f\n", c, c.Area())
}

Building it with go build:

$ go build

And finally let’s test it out by running the resulting binary:

$ ./hello
Area of Circle(5.00): 78.50

Congratulations! 🎉

Now you’re a Gopher!

That’s it! Now you know a fairly decent chunk of Go. Some (pretty important) things I didn’t cover include:

• Writing unit tests, writing tests in Go is really easy! See testing
• The standard library, Go has a huge amount of useful packages in the standard library. See Standard Library.
• Goroutines and Channels, Go’s builtin concurrency is really powerful and easy to use. See Concurrency.
• Cross-Compilation, compiling your program for other architectures and operating systems is super easy. Just set the GOOS and GOARCH environment variables when building.

For more details, check the latest documentation, or for a less half-baked tutorial, please read the official Go Tutorial and A Tour of Go.

Twtxt is on Reddit!

The decentralized micro-Blogging Twitter™-like platform I created back in Aug 2020 of this year; jointwt/twtxt that powers Twtxt.net (see also JoinTwt); got posted and talked about on Reddit today! 😱

Hey @manton I got this email today from a registrar claiming that my prologic.blog domain will expire soon. What’s up with that? Doesn’t...

Hey @manton I got this email today from a registrar claiming that my prologic.blog domain will expire soon. What’s up with that? Doesn’t your infra handle this automatically?

Hey @DavidAnson just came across your writeup over at github.com/DavidAnso… ! Very nicely done! 👌

Hey @DavidAnson just came across your writeup over at github.com/DavidAnso… ! Very nicely done! 👌