Robert Hafner is a Principal Engineer based in Chicago focusing on machine learning operations, distributed applications, infrastructure, and security. This blog is a running journal of projects, tutorials, and random ideas that pop into his head.
✍️Write rieview ✍️Rezension schreiben 🏷️Get Badge! 🏷️Abzeichen holen! ⚙️Edit entry ⚙️Eintrag bearbeiten 📰News 📰Neuigkeiten
In reply to <a href="https://blog.tedivm.com/open-source/2023/02/simple-multiprocessing-with-quasiqueue/#comment-98">Anonymous</a>. If you were to chain queues like that you would start with one instance of a writer with a group of readers, and then each of those readers would have their own writer with it's own readers attached. So you wouldn't have two instances, you'd have n+1 instances of writers where N is the number of readers attached to the first process. What's the use case for that?
6.5.2024 00:37Comment on Simple Multiprocessing with QuasiQueue by Robert HafnerRobert Hafner at the Open Source Summit: Bringing CI/CD Practices to Machine Learning with MLOps
27.4.2024 01:00Open Source Summit: Bringing CI/CD Practices to Machine Learning with MLOpsHow would QuasiQueue work with 2 chained queues? That is: Producer --> Queue 1-->Consumer Tasks Group 1-->Queue 2-->Consumer Tasks Group 2 ? Would one make two instances of QuasiQueue and connect them somehow?
16.2.2024 23:21Comment on Simple Multiprocessing with QuasiQueue by AnonymousSQLAlchemy is a powerful tool that lets Python developers define their database schemas using simple classes. I made Paracelsus to turn those SQLAlchemy defined databases into diagrams that can be used to visualize the schema.
8.1.2024 14:30Visualizing SQLAlchemy Databases as Mermaid and Dot Diagrams using ParacelsusRobert Hafner on The IaC Podcast.
21.12.2023 14:00Podcast: IaC in depth with Robert HafnerJShrink is a Javascript minifier written in pure PHP. It allows web applications such as Magento and Matomo (as well as thousands of other projects) to shrink Javascript code on the fly. In 2014 JShrink reached stability, and shortly after that hit the milestone of 100k total downloads- today JShrink has over 21 million downloads,...
6.3.2023 16:00JShrink reaches over 21,000,000 installs and releases v1.6!Rob's Awesome Python Template is a highly configurable CookieCutter template that's packed full of best practices for Python Projects.
3.2.2023 16:59Rob’s Awesome Python TemplateQuasiQueue is a MultiProcessing library for Python that makes it super easy to have long running MultiProcess jobs. It handles process creation and cleanup, signal management, cross process communication, and all the other garbage that makes people hate dealing with multiprocessing.
2.2.2023 17:30Simple Multiprocessing with QuasiQueueThe action-python-lockfile-update project is a Github Action, freely available for anyone to use, that uses pip-tools to rebuild your requirements.txt files on a schedule.
1.2.2023 17:30Automatically Updating Python Requirements Files with Github ActionsIntroducing DapperData, a CLI for formatting JSON and YAML that works well with Python code bases.
31.1.2023 15:00Introducing DapperData for Formatting YAML and JSONHi Robert. Thanks for this excellent and detailed blog post. I managed to adapt it to our setup perfectly. Saved me tons of time. Thanks!
20.6.2022 12:29Comment on Using Github Actions OpenID Connect to push to AWS ECR without Credentials by Klaas-Jan WierengaUpdate: I think I put everything where it should go and got it at least terraform planning locally. Basically main.tf goes in the root ( which in my case is a "terraform" folder) and everything else goes under a modules/repositories folder (modules also being under the "terraform" folder) and the source referenced from the module is './modules/repositories' - I also did have to fix a typo or two (locals.repositories should be local.repositories in main.tf, and in github_iam_role.tf I had to change the identifiers line to identifiers = [var.oidc_arn] to match the passed-in module variable)
31.1.2022 20:09Comment on Using Github Actions OpenID Connect to push to AWS ECR without Credentials by haussenfefferIs there a folder structure that should be used when using these files, or a specific terraform version? When I put all the tf files in the same folder and I try to run terraform init (with the latest version, 1.1.4), I get Initializing modules... ╷ │ Error: Invalid module source address │ │ Module "repositories" (declared at main.tf line 14) has invalid source address "": invalid source string: . ╵ Thanks for the great article!
31.1.2022 18:56Comment on Using Github Actions OpenID Connect to push to AWS ECR without Credentials by haussenfefferThanks Robert, this article was really helpful!
23.12.2021 00:45Comment on Using Github Actions OpenID Connect to push to AWS ECR without Credentials by Paul C.In reply to <a href="https://blog.tedivm.com/guides/2021/10/github-actions-push-to-aws-ecr-without-credentials-oidc/#comment-23">dennis</a>. That's the <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html" rel="nofollow ugc">AWS Account ID</a>. Each account has a unique one, and you should put yours in there.
15.12.2021 17:51Comment on Using Github Actions OpenID Connect to push to AWS ECR without Credentials by Robert HafnerI don’t understand how you get this " AWS_ACCOUNT_ID: "999999999999" " . can you give a insight on this.
8.12.2021 04:55Comment on Using Github Actions OpenID Connect to push to AWS ECR without Credentials by dennisUsing the AuthorizedKeysCommand OpenSSH option to pull your SSH Keys directly from Github.
19.11.2021 14:08Telling OpenSSH to Pull Keys from Github with AuthorizedKeysCommandIn reply to <a href="https://blog.tedivm.com/guides/2021/10/github-actions-push-to-aws-ecr-without-credentials-oidc/#comment-21">ordisius</a>. That's actually because the Role is being created inside of the module that creates the ECR repo, so the resource is being passed in as a variable. This allows us to reuse the OIDC resource while still having a module for the ECR repo, which in turn lets us use the <code>for_each</code> resource when creating our repositories.
11.11.2021 16:23Comment on Using Github Actions OpenID Connect to push to AWS ECR without Credentials by Robert Hafnerawesome post!!! I think there might be one typo in the terraform code <code>var.openid_connect_provider.arn</code>, probably started with that value as a variable, but then switched to the resource attribute
11.11.2021 03:59Comment on Using Github Actions OpenID Connect to push to AWS ECR without Credentials by ordisiusMulti-Py publishes container images for popular Python packages targeting AMD64, ARM64, and ARM v7, on Ubuntu and Alpine.
10.11.2021 14:22Multi-Py: Multiplatform Container Images for Python Packages