in case you’re thinking “I must have read something similar before”, you guessed right: I’m not the first one to write about this topic. Others have done this before me. Many, others, in fact, like this one, this one, this one, this one, this one, or, famously, that one, which this post is kind of based on (thank you for laying the groundwork, Elio!). While being cool, there’s a few issues.
But listen, there is good news.
I WILL FIX ALL OF THIS (mostly)
In this blog post, I will show you how you can:
Big goals. But before that, a little bit of a disclaimer as to 1) who this blog is written for, 2) what it is we’re doing and 3) why we are doing it.
This blog is for people unfamiliar with docker, or people familiar with docker who never used rocker or tried to use it for reproducibility. This means, I will go a bit into operational details with docker and its prerequisites—something people familiar with the program might perceive as a bit too “goo goo gaga” for their taste. But this is fine. Just take what you need and leave the rest. The technique I outline will enable your product to be fully computationally reproducible. If you’re in science, you might know the “data available upon request” meme. It’s something authors write in their paper when they want to pretend to be open and inclusive and all that, while either having lost their data, made it up in the first place, or milked it in unethical ways. Of course, sometimes they are just plain lazy—which I suspect is the broad majority. Regardless: With computational reproducibility, we can prevent that and at least see how the authors got to the results they report in the article based on their data and code. Computational reproducibility therefore mandates open data and code, but also means we need to make the result easily reproducible for anyone interested in doing so. This is what this post is about. Note that computational reproducibility is not the same as methodological reproducibility, or even integrity. All this blog post will tell you is to make your analysis reproducible for others. Not your experimental setup, not your survey, just the analysis. Reproducibility does not mean your result isn’t based on fraud, incompetence or misused analysis. So don’t take this as an excuse to manipulate your way into a publication while claiming moral superiority because your fraud is reproducible. That is not how it works. But for now, back to the topic at hand.
A bit of Linux knowledge (Linux users may skip this)
“Why Linux when I use $GARBAGE_OS for my analyses?”, you might ask. Well, I will not judge you for liking garbage. Hell, I’ve lived off of garbage for a long time. But docker basically runs on Linux. You might have heard about Docker for Windows, but that runs on the Linux subsystem for Windows (WSL), not Windows itself. So here’s everything you need to know about Linux to use docker. Linux runs basically on the command line, and with docker, it’s best to get used to that. While Docker desktop exists, I consider it enshittified UI trash; personally, I find dealing with the command line clearer and, frankly, easier. The command line lives in the terminal, in Windows that’s the cmd or powershell thing. You start it and type in commands.
The commands you need to know with docker are quite simple actually. There’s cd, short for change directory. It takes a path after it as an argument, which is the destination where you want to go and sets your working directory of the command line to that folder. You can input any valid folder path into cd, it will take you there.
#this changes the working directory of the command line to the mbess-project folder
cd home/documents/r-projects/mbess-project
Another interesting command is cp. This literally means copy paste and takes two file paths as an argument: The origin and destination paths. Note that if the filename in the destination already exists, it is overwritten without asking. If there is no source file by that name, it is created without asking (i.e. an empty file spawns at the destination).
#This copies the file "text.txt" from documents to downloads
cp home/documents/text.txt home/downloads/text.txt
Then you might want to know about ls. This command lists all files and folders in your current directory (or another specified directory) and gives you a broad overview what to get.
#gives all contents of the mbess-project folder
cd home/documents/r-projects/mbess-project
ls
You can view files in the command line with cat. You can also insert a path to a file that is not in your current working directory.
#this prints the file contents to the command line
cat filename.r
Docker/podman
Docker and podman are container software. Containers are cool, because they are like virtual machines, just smaller and less intensive for your digital abacus. They can provide a backend running some operating system, and also provide pre-installed software that runs on that operating system. You can detach and attach them from your machine with one command. Isn’t that handy?As an example, docker was used to deploy images for phishy mailbox, a phishing research software Thorsten Thiel and I created. You plug it in and use it in the browser, no matter what your operating system behind the browser is. If you’ve ever been barred from using software because your system can’t run it, think about how cool it is to not need to care about that.
Docker is the more seasoned version of the two, and is somewhat proprietary, but free to use for non-profit things or private persons—it also (still) scales better with really big applications and clusters than podman. Podman is a free and open source container software, which is becoming more and more popular among the community. It is highly compatible with docker and basically does most of the same things. I will focus on docker in this post, because I personally haven’t dabbled with podman much so far, and don’t feel too confident talking about it. But eventually, it is my goal to implement this approach fully with podman as well.
Installing Docker
You usually always need admin rights to install docker, this is true for any system if I remember correctly. This most likely means contacting your IT department if your IT is somewhat centralized and knows what it’s doing, so keep this in mind. On Windows, the easiest way to install docker is through Docker Desktop (and then forget about docker desktop). On Mac you will need homebrew or something—you may google this, I am not a Mac user and do not intend to burden my brain with knowledge of that system.
Once installed, you immediately (and trust me, this is important) need to disable root needs. Docker usually needs admin privileges (called “root”) to run, and you really don’t want that. Firstly, because your admin may not trust you enough to execute random containers (possibly containing malware) with admin privileges, and also because it’s a big nuisance to always enter admin credentials for docker to work. Consult this and ask your favourite IT person (which is definitely not me) to help you if you need it. Don’t be shy, they will be glad to see you want to learn things.
Docker has a few basic commands we need to know for our workflow:
Rocker
Rocker is a family of pre-configured docker images. The base version comes with r installed, but there are also versions with Rstudio (server) and the tidyverse pre-installed. This is highly useful, because it gives us a head start in our endeavour.
renv
renv is an R-package that will install your used libraries into your current project folder and lock their version. This means, you have a static version for your libraries, and will not cause dependency or downward compatibility problems with your code down the line if they happen to update on your system at some point. This also ensures computational reproducibility in our docker case.
pak
pak is an R package which kind of flies a bit under the radar, but is extremely useful for our specific workflow. Since our base Docker image (rocker) uses linux under the hood, we need to make sure all our R packages run on linux, and a specific version of linux at that. Some R packages have so-called system dependencies, i.e. need other software on the system to work properly. While those system dependencies may be easy to gather from the documentation of a single r package, it gets quite convoluted once you handle many packages which themselves depend on other packages. pak can do all this for us. We can give it a vector of package names and it will output all dependencies we need to make the entire stack work, including dependencies. If we specify the linux distribution, we can even get a command for installing those dependencies. That is, as we nerds say, cool stuff.
There are basically two ways to achieve your goal, depending on the state of your project and what you want to achieve. You can:
Start working with a container from scratch and build it slowly. While this is an approach some recommend, it also has some pitfalls—you risk bloating the container with unnecessary installs and might have to spend more time debugging it in the end anyway.
Making an ephemerally usable, reproducible image of your finished project from a pristine template. The ephemeral part means that people will be able to execute your analysis from scratch without altering the image itself—no results data will be stored in the image itself, even though temporary data might occur. People will still be able to reproduce your work, even copy results to their own local folders. For computational reproducibility, this is all that matters. Your results are in your paper, and your methodology is there also. This is the approach I recommend and will detail here.
The process consists of a few steps: First, we set up renv with our project, then we find out the necessary system dependencies for rocker with pak. Subsequently, we create a Dockerfile, detailing how our image is to be built by Docker. Finally, we will build, test, and push the image. Let’s rock(er)!
renvInitially, you need to set up renv. To do this properly, we need to ensure a few things: Firstly, the package itself needs to be installed on your system—you can get it from CRAN as usual in RStudio. Then consolidate all your R files into one folder. Once this is done, set your working directory to this folder. Since renv sometimes bugs out detecting packages, make sure you call every library you use at least once directly with syntax like library::function() e.g. ggplot2::ggplot(...). In fact, make it a habit to do this every time you use a function of a newly used library for the first time. If you use the rocker tidyverse image to build, you can call tidyverse as a library. If you don’t, you cannot do this, you will need to call all the single packages manually, e.g. ggplot2, broom, purr, readr (!) etc. This ensures that renv will discover the packages you loaded, which is what we will do next. Call renv::dependencies() in your console and renv will show you which libraries are used within all R files in your working directory. I personally always check this against my library vector, which I highly encourage you to do as well. I usually call libraries in bulk from an initial vector in the form of the following:
pkg <- c("package1","package2",...)
for (i in pkg){
library(i, character.only = T)
}
This comes in handy when we will use pak in a bit. For now, the dependencies command should show you all the libraries in your pkg vector. If not, you either might have not called one of the libraries and renv didn’t recognize it—this can happen with code in function definitions sometimes—or you have a library in your pkg vector that you are not using in the code (anymore). If everything checks out, you may call renv::init(). This command will lock your loaded libraries to their current version and create an renv.lock file. This is the thing we want, because it can recreate our environment in the docker image. But first we need to know which dependencies our packages have on a system level, this is where pak comes in.
pakRemember that I said docker runs on Linux? Well, some packages require a certain help from the outside of R to work. Thankfully, pak helps us with this. We install it from CRAN and call the pak::pkg_sysreqs() function, which we can use with our pkg object. This will give us all the system dependencies as well as a command to install them in the docker image. Save this somewhere, we will need it. As an example, below is the code for a project using only base R and the MBESS library. Remember that for renv to work properly, always include “renv” and “curl” into your package vector, and the pkg_sysreqs() command.
pak::pkg_sysreqs(c("MBESS", "renv", "curl"), sysreqs_platform = "ubuntu-20.04")
#Output:
# ── Install scripts ───────────────────────────────────────────────────────────────────────────── Ubuntu 20.04 ──
# apt-get -y update
# apt-get -y install make cmake pandoc libcurl4-openssl-dev libssl-dev
#
# ── Packages and their system dependencies ──────────────────────────────────────────────────────────────────────
# curl – libcurl4-openssl-dev, libssl-dev
# minqa – make
# nloptr – cmake
# OpenMx – make
# RcppParallel – make
# rpf – make
# StanHeaders – make, pandoc
The thing we want is the install scripts section. This is what we need to install to our docker image to make our code work. About the ubuntu version: You needn’t worry. We can find that out manually, it’s a great exercise in getting to know how rocker works, I’ll tell you how to do that at the end of this blog post. For now, we have all the info we need.
Docker images are built using an instruction file called a dockerfile. The docker engine will look at the file and do everything we detailed there to build the image. To create a dockerfile, we open our text editor of choice, type in the necessary stuff and save it as “Dockerfile”. No file extension, nothing. Just “Dockerfile”, in the folder where your data and code are located.
For our example project, we have our R folder as a target, containing an analysis file called analysis.r, an arbitrary amount of data files in the data folder within the directory, our renv.lock file, a README.txt file showing people how to proceed once they initially ran the image, the renv folder, and our Dockerfile.
Below is the code for the Dockerfile. I will explain what each section does in the following paragraphs.
FROM rocker/rstudio:4.5.2
LABEL maintainer="email@email.com"
## Install system dependencies
RUN sudo apt-get update
RUN apt-get -y install make cmake pandoc libcurl4-openssl-dev libssl-dev
WORKDIR /etc/rstudio
#copy rstudio initial instructions to override defaults
COPY rsession.conf rsession.conf
WORKDIR ~
#install renv
RUN R -e "install.packages(c('renv'), repos = 'https://cloud.r-project.org')"
WORKDIR /home/rstudio/mbess-project
# Copy data and scripts
COPY data data
COPY README.txt README.txt
COPY analysis.r analysis.r
# copy further things if they exist....
# Copy R environment and restore packages
COPY renv.lock renv.lock
COPY renv renv
RUN R -e "setwd('/home/rstudio/project-name');renv::restore()"
Let’s start at the top.
FROM rocker/rstudio:4.5.2
LABEL maintainer="your-email@your-email.com"
This instructs docker to use the rocker/rstudio image with the specified R version, in my case it’s 4.5.2. The cool thing is that you can just use the version you are working with, it doesn’t really matter how old. Rocker has got you covered. To inquire your current R version from your project, use the R.Version() command from base R.
The label is just an info for people who might need support or have questions to get back to you, so insert your email address in there. If you want to submit this to peer review, think about anonymity here—set up a new email for it somewhere under the project name or something.
## Install system dependencies
RUN sudo apt-get update
RUN sudo apt-get -y install make cmake pandoc libcurl4-openssl-dev libssl-dev
Remember the install command we got from pak? This is where we need it, because without it, our renv code won’t run. In our little mini project using MBESS we had apt-get -y install make cmake pandoc libcurl4-openssl-dev libssl-dev, so that is what we’ll use. The “sudo “ part tells linux to use admin rights to install the thing. It is usually not necessary, as you already have admin privileges for rocker building, but just in case, we will use it here (it is technically considered bad style in dockerfiles, but sometimes one runs into issues, so I will keep it). The update command ensures that our system can find the necessary packages to install.
The next step ensures that Rstudio server in the image starts up loaded in our project directory so all we have to do is click on a file and source it. For this, we need to give RStudio a pointer to a default folder to load into—this is never set automatically. Digging though the documentation got me here, showing which parameter we need to change. We then make up a new rsession.conf file in a text editor with the following content:
# R Session Configuration File
session-default-working-dir=/home/rstudio/mbess-project
This tells RStudio to open new sessions always in the mbess-project folder. Obviously, we can rename the folder to whatever we want, we just need to be consistent with the naming in the dockerfile. We save it into your R folder that contains the dockerfile.
WORKDIR /etc/rstudio
#copy rstudio initial instructions to override defaults
COPY rsession.conf rsession.conf
The above code then tells docker to switch its working directory (very analogous to R) to the internal etc/rstudio folder in the rocker/rstudio image. There, an rsession.conf file already exists, albeit empty. The COPY command (similar to cp) replaces the empty file in the rocker/rstudio image (right side) with the one you saved in the directory on your local machine (left side).
WORKDIR ~
#install renv
RUN R -e "install.packages(c('renv'), repos = 'https://cloud.r-project.org')"
After this, we tell docker to switch its focus back to the home directory (~) and install renv. We do this with a BASH command, which tells us to run R and then install the renv package from a repository.
WORKDIR /home/rstudio/mbess-project
# Copy data and scripts
COPY data data
COPY README.txt README.txt
COPY analysis.r analysis.r
We then switch the docker working directory to our mbess-project again—if it does not exist, docker will create it, so we won’t worry too much about that. We proceed to copy the local data folder, analysis file and a the README file from our local machine (left) to the docker image (right). If there is other stuff necessary to copy, we could also just insert file paths. I personally just find it easier to do it from one folder, especially since I have an easier time curating my data and scripts beforehand.
# Copy R environment and restore packages
COPY renv.lock renv.lock
COPY renv renv
RUN R -e "setwd('/home/rstudio/project-name');renv::restore()"
The last part of the code is the most critical. This will copy all the renv-relevant files and folders, then run r, set the R working directory to your project and call the renv::restore() function. This may take a while, because this installs all your specified packages and their dependencies into the project folder, with versions identical to ours. This often means that we need to compile those packages from source, i.e. we can’t just “install” them in the regular fashion, but need to build their installed versions from scratch code, which takes a while. Granted, most packages will build in less than 30 seconds, but some may take minutes; as an example, OpenMx is one of the nastiest ones…if you run SEM-code, expect some significant building time.
This is about it. Docker does not need to know more. Commence the building!
This is a rather simple process. We switch to the command line and, if we set up docker like described above, we can just change directory to your R project folder including the dockerfile and kick-start the building process.
cd home/r-projects/mbess-project
docker build . -t k4tana/mbess-image
Note that naming convention in images is “creator/image-name”. As creator, you can just use your dockerhub/github handle, or whatever fits. Docker will tell us if it runs into issues during building. Mostly, it will just be small typos in file names in the dockerfile or things of this matter. Sometimes, the build will fail to install packages due to missing system dependencies. In that case, we need to check renv::dependencies() again, sometimes a package might have gone unnoticed by renv.
Before publishing anything, we need to test the image. This means: Running it on our machine to see whether it runs as intended. To run the image, we need the following command in the command line.
docker run --rm -p 8787:8787 -e DISABLE_AUTH=true -v /home/rstudio/mbess-project k4tana/mbess-image
This tells docker to run our image (k4tana/mbess-image) on port 8787 in ephemeral mode (–rm), meaning all the things we do with the image while we run it will not be saved within the image, so that it starts up fresh next time. It also disables authentication so we don’t have to type in login or anything—we are doing open science and have no secrets, right? It also tells us which folder/volume we would like to mount within RStudio-server (-v).
After this, we simply switch to the browser of our choice, type localhost:8787 into the address bar and watch RStudio Server start up in the browser, ready with the code and data. It is highly recommended to run all scripts once fully to verify they work. Sometimes there are issues with parallelization and memory allocation, which need fixing.
Once all runs and works, we can publish our image. The easiest way is to push it to dockerhub. For this, we create an account there if we don’t have one already. We then authenticate via the command line with docker login and follow the instructions we are given. Finally, we can push the image to dockerhub with docker push k4tana/mbess-project. If it complains about a missing repository, there might be a need to create one on the dockerhub website prior to pushing. In this context, it is good to know that renaming images is easily possible with docker image tag old/name new/name.
Again, we test our image by pulling it with docker pull k4tana/mbess-project, to our machine and also a different one, preferably with a different OS to see whether it works. If it works and executes, we are officially done and can pat ourselves on the back (shoulder mobility!), as we have finally made our project computationally reproducible.
Building shenanigans
Should your project fail to build, you can build again after improvement of the dockerfile, with low penalties. Usually, docker caches (saves) your process until it breaks, so you don’t have to repeat anything until the first command that was changed. If you want to build from scratch, you may clear your cache with docker builder prune --all and then confirm.
If you built an image that contains an error, you can delete it using docker rmi imagename and build anew.
Rocker Ubuntu Version
Finding this out is kind of hard—the documentation itself doesn’t make it clear. To get the easiest result, pull the rocker/rstudio image you want and execute it without building your own stuff on top. Use a run command for the image, then in RStudio Server, switch to the terminal (not the console!) and run lsb_release -a. This will tell you the version of Ubuntu you need to specify in the pak command.
Using Github or Codeberg
Of course, you can also use the systems Github and Codeberg provide to host your images. Just be aware of anonymity, should you want to push for peer review. You can find decent tutorials on how to push your stuff there online (except Codeberg, their documentation is at time of writing awful regarding that, but AI might help you here, for once).
Notes on renv
renv, while sometimes unreliable with library detection, isn’t always at fault. Sometimes, package developers make mistakes, too. A thing I sometimes see is package maintainers being hesitant to declare something a hard dependency instead of a suggestion, even though it practically is one. renv then will not include the packages in the renv.lock file. The end result is your build failing due to lacking dependencies, which is the worst possible outcome—build times can be up to 40 min for big projects and renv installs of packages are the last part of the building process. There is a workaround for this, however, detailed here.
Stopping Containers
If you are running a docker container but want it to stop to save resources, the easiest way is to visit the command line with the running process, and press CTRL+C. This closes the process.
I hope you got a glimpse into the process of using Docker and R to make your projects computationally reproducible. Granted, it is not the technically best approach—I believe what Bruno Rodrigues has done with NixOS and Rix is superior in terms of streamlined reproducibility. Yet, I fear that his approach is too technical for many scientists whose skills are better used in the lab or interviewing people rather than learning a new operating system just to get their papers out. Docker is, in my opinion, a good compromise because it is easy to learn to get the job done, works on any system and is free to use. With the advent of podman and its continued development, we will also see a completely FOSS version possibly taking the reigns soon. While some argue that Docker is an added dependency for projects, I argue that sometimes a dependency is worth it, if it makes a process more usable for most people involved.
Let me know where you stand on the matter, whether you found this helpful, or if there is anything I could clarify.
Cheers.
]]>“AI” is a bad term.
I prefer using Machine Learning (ML) for the “old” algorithms like random forest, neural nets or SVMs, GANs etc., and “transformers” or “LLMs” for the newer model-combinations underpinning chatgpt, claude, llama et al. While the former models are all relatively old – neural nets were invented as a “perceptron” by a neuroscientist in the 80s – their explosion happened with the computation revolution from 2000 onwards. Cluster computing, the mass adoption of the internet and easy cash due to the bull market before 2008 made it easier to gather training data, build infrastructure and develop models. Since then, models like decision trees (CARTs), SVMs and such are considered “off the shelf” models, with a rich knowledge base behind them. You can train such models on an average computer, with decent results. And they are everywhere. When you swipe your credit card somewhere, one of these models will predict the risk of you defaulting on the credit in milliseconds and either block or greenlight your payment. When you click on something on Youtube, a system will integrate the data and recommend you new videos based on some optimization algorithm based on an off-the-shelf system. These things are computationally efficient and easy to implement. “AI”, if you will, is everywhere. Has been for close to a decade now.
Now, transformers are a different beast altogether. Their main hallmark was initially proposed by Google. The idea behind it is basically simulating a large net of neurons and making it computationally efficient, giving it the possibility to weigh certain parts of the input more or less than others, and cross-referencing data better through a “knowledge map”. This leads to a similar mechanism our brain implements: We heavily filter our sensory input and only retain the important stuff. By doing this, the machine can be more efficient in dealing with information and give you a better result. The output of a transformer is purely statistical: Given input A, it will generate an output B that has, according to their inner wiring and training data, the highest probability of fitting the input. If you ask it, “How are you?”, it will answer “Fine, how are you?” back. It fits statistically, because its training data showed this pattern of communication very often, making it likely that “Fine, how are you” is the best response to your question. That’s basically how chatgpt works. For context, this is not how all transformers work, there are also other ones for visual generation, for translation and other things. I will focus on chat and image transformers in this piece, since those are the ones currently talked about the most. So, now that you have a bit of an understanding, let’s get to the most uncomfortable stuff.
The attentive reader will have noticed that I am, right now, writing about “AI”. And I will not exclude myself from belonging to “most people”. Thus, I will preface the following text with a disclaimer: I’m not a “trained” data scientist (although data science is what I’m hoping to do after my PhD), I have no in-depth knowledge of how the large transformers and cutting-edge models work. That being said, I did have above average training in “AI” and statistics during my studies and continue to learn more about the subject due to my job and just because the topic is interesting. This means: Take everything here with a grain of salt. I think I know what I’m talking about most of the time, but the burden of falsification is on you, and everyone can make mistakes (especially scientists, and especially me). Alas, let’s start with the most obvious fact about “AI”.
“AI” is hype tech
There seems to be no middle ground when it comes to opinions about “AI”. I find this funny, because this is basically what any hype technology has ever evoked in people. Extreme polarization is a hallmark of new tech. Here’s a few examples: Train rides were once suspected to “injure the brain” when they became popular. Hell, even bicycles caused adverse reactions when they became a serious means of transport for women. In Germany, there is a famous quote from former Telekom CEO Ron Sommers in 1990 stating that the internet will probably never become popular, being a mere playground for nerds. All of this has aged like fine milk and is a reason to disregard tech hype (and anti-hype), or at least adjust one’s expectations. Right now, we as scientists and societies are dealing with the question whether social media is bad for people, and the promoters of this hypothesis engage in…worrying logic. Regardless: most new tech will die (remember Google Glass?) or be rather irrelevant in the grand scheme of things. I see the latter happening to “AI”. And all sides discussing the issue seem to dig deeper into their opinions, some nonsensical, some hypocrisy, some valid.
“AI“ is based on intellectual property theft
This is what you will hear a lot of artists saying about visual “generative AI”, because companies scraped every available source of information on the internet for transformer training data, including artists’ promotion pictures on Instagram. Especially art seems tricky to evaluate, since intellectual property does not always apply to art – depending on the region of the world you’re in, taking a picture of a sculpture in someone’s yard and selling the picture would not be IP theft or very much so. Thus, there might be some merit to the theft claim, especially considering the rules being heterogenous. It is undeniable that the companies never asked for approval, and certainly never compensated anyone. While this might be immoral, it need not be illegal – although it seems like it ultimately probably was. The damage, however, is done, and getting cats back into bags is very difficult. Once something is out there, it’s out there. The internet will probably not forget (even though it should). And whether you can recreate pictures from a specific artist through a model trained with their art is also – to my knowledge – unclear as of now. As a side note, I find it curious that especially independent artists, who have long since criticized IP legislation and copyright practices of big players, especially in the music industry, are now the ones demanding protection from the regulations they once despised. While I am generally sympathetic to their claims and do not want to see independent music die a gruesome “AI-slop” death (“AI slop” is a term for “AI”- generated low-quality content in whatever form), the irony of the situation tickles my funny bone.
We will develop “Artificial General Intelligence (AGI)” in [insert time frame]
AGI firstly is a vague term, at best. Some people claim it is supposed to surpass humanly possible intelligence, others just say any system that is profitable is AGI. There’s a LOT wrong with both definitions. The first one is impossible to measure. Firstly, we ourselves currently do not know how far human potential goes. Let’s look at IQ as an example. IQ is the best tool we have today to measure important mental tasks. The highest IQ values today lie close to 300 (average humans have 90-110). Keep in mind, this data is from people we tested, which are few and far between on a global scale. But the bigger issue is: AGI is supposed to be way above our level. To measure this, we must design a test that measures ability we cannot fathom. But a test needs pre-conceived answers to evaluate it. If we cannot imagine what it is we want to test, we cannot test it. It is, therefore, unscientific to make such a claim, at least if we trust the “mainsteam” scientific method. It makes for cute philosophical arguments, but is ultimately marketing speak. The most we can test is “better than all humans before”. Is that AGI? Or is that another human-like machine? Answer: Probably the latter, because, you guessed it, we used human data to train it. And that data is noisy. If you really think that social media posts, video descriptions and texts are a good representation of thoughts, think about a picture or movie scene and try to describe the image or sequence in your head in text. Good luck with that.
All of this means to me: At least with the current type of models, AGI will not happen. I say this with confidence, also for a few technical reasons. Lets just imagine AGI was something measurable and real for now. Even though the current models are sold as “PhD-level intelligence” (which means nothing in terms of intelligence), they cannot even come close to human function. You will realize this once you give them a fairly complex task, for example coding. They will break down at some point and produce nonsense that clearly is not based on analysing and compartmentalising a problem, but spraying random garbage all over the script in order to increase statistical fit. It is akin to a trapped, panicking wild animal. It might “know” a lot and be able to search the web for answers, but knowledge and intelligence, while related, are not the same thing. On top of this, to produce mere human-level abilities, a model must be able to:
The straws that break the camel’s back here are point 3 and 4. We cannot live-update these models, because right now you need to re-train the entire thing. And training is expensive. A large BERT (transformer) model can consume about 1500 or more kWh for a training cycle. To gain some perspective, let’s compare this to a human brain. Our brains run on about 20 Watts per day, which means its energy consumption is about 0.8 Wh. If we used all the energy from one training run of this BERT model for food, a human brain alone could live more than 214 years. Considering that the brain uses about 20% of the body’s daily energy needs, a human could fully live off this energy for more than 40 years. The brain dynamically updates stuff over 40 years with the same total energy consumption that one of these models uses to consume and process very large amounts of data once, with the weird result of recommending glue on pizza. Compared to the machine, we ingested more modalities, parallel computed them better, filtered noise better, built a huge repertoire of skills – language, motion, recognition, sensors, memory – and got rid of inefficient connections. Indeed, the human brain actually reduces connections and kills neurons until you’re about 30 years of age, to become more efficient. And considering that the basic properties outside of acquired job-related skills take less than 20 years, this is bonkers. The machine cannot compete energy-wise. It cannot compete in the breadth of skills. Saying we will develop AGI by just tweaking those models is psychotic.
“AI” will replace or boost social interactions.
To me as a psychologist, this is an especially interesting one. Mark Zuckerberg, a man famous for looking like a robot in congressional hearings, recently stated that his goal is to equip people with “AI” friends to boost the friend count of the average American over 3. To anyone with a brain (and friends), this clearly shows that Mr. Zuck does not know what friends are. Which seems realistic, seeing as the only friend this person seems to have retained is now his wife, and I’m not sure whether those two are still on friendly terms. In any case, being lonely is not cured by “knowing more people”. You’re not looking for a number, you’re looking for meaning and connection. This is pretty basic “being a human 101”, but that seems to elude the Silicon Valley oligarchs ever so slightly. Also, “AI” has terrible memory. I can trash talk my friends based on a random incident in 2010 which we all remember fondly as the time where we collectively fucked up. Your “AI” friend might get the dates wrong, if it didn’t run out of memory before. Related (stay with me, I’ll get to the point quickly): para-social relationships with influencers only work, because we know that the other side has a personality – or at least something like it, depending on the influencer. You don’t know what chatgpt is. It changes personality and interests all the time, depending on the input. That is something humans really do not like. In general, we prefer consistency. Sure, some people might like it, and I as an introvert see value in not needing to interact with humans. But I believe the majority still seek experiences with real humans, mysteries which we need to solve to understand each other and the flame of social warmth that ignites once we do.
“AI” will train you to think less and become dependent on it
This is something relatively new. I’ve been reading a few things on social media lately pointing in this direction, the most prominent in German tech (and security) circles being this German blog post by Mike Kuketz, a well-known name in the German infosec and privacy community. I respect Mike heavily for the work he does – if you want to learn something about privacy and security, be sure to visit his blog, he has lots of good tutorials and information on there. Unfortunately, this blog post was not his best work, as it falls into the category “Computer scientist talks about the human brain and behaviour” – which is a massive red flag to me as a psychologist. Maybe I’ve read too many HCI papers, but when I first opened this blog post, I experienced a sense of dread which I usually only get with the mentioned HCI papers. And, apparently, it was not a false alarm. Mike argues that humanity will forget how to think for themselves and become dependent on (“AI”-) technology due to heavy “AI” use. In a scenario reminding me all too much of the movie “Wall-E”, Mike predicts our future as lazy drooling “AI” slaves, chasing convenience over effort. Freely translated, Mike says: “People who early on learn to outsource responsibility for decisions, language or thought will never learn to recognise responsibility. “AI” then will not be used as a tool, but as a proxy: For knowledge, for experience, for decision-making skills.” This is a stark claim. But it’s not new. The same claim was made about TV in the 1950s, shortly after the device became more widely used in homes. Didn’t we talk about hype technology? Granted, fearmongering like the old times need not be wrong, after all, you might be right to be concerned (nukes, anyone?). However, this case is different. There is nuance to everything. Let’s address these things step by step.
Now, while I’m basically at least skeptical about all the claims I’ve described above, this need not mean that “AI” is harm-free, loosey-goosey joy in a black box. Let’s dive into the dark, deep end.
There is, as always, a caveat to everything. “AI” does pose real dangers and real problems, to society, individuals and organisations. While I try to avoid demonizing technology for reasons already explained, there is potential for actual harm using those tools. I’ll just list the ones I think are most relevant.
Fuelling climate change
As I have shown above, a single training run of a BERT model can consume quite the amount of energy. But: This is just a BERT model. Chatgpt is a different beast, not just one single model. Training this probably costs an order of magnitude more power, 1k+ Mwh. And that’s not all. After all, if you use it, it also needs energy. Usage represents the lion share of power draw for such models. Depending on who you ask, an average prompt to Chatgpt can consume between 0.2 and 3 Wh of power. This obviously does not sound like a lot to most people. “What are 3 kWh even?”, you might ask. I’ll tell you: Know these small drink fridges that can hold 10-15 bottles? 0.2 to 3 kWh per prompt means 40-600 prompts to “AI” will basically draw as much power as the fridge consumes in a year. Now while I believe the 0.2 kWh estimate is tad biased (it’s from an “AI” company), 600 still isn’t that much. Some people use “AI” heavily, generating hundreds of prompts every day. There are estimates out there showing the entire power requirement for worldwide “AI” tech is akin to a small country like Switzerland. Most of that energy does not come from renewables, meaning that we add another country’s worth of CO2 emissions (worst case) to our world’s carbon weight anually. Considering the energy we already waste for blockchain BS, this “AI” craze, eventually, will drive up costs for everything due to climate mitigation costs, while also being harmful for the environment and our planet.
Misinformation/Disinformation
“AI” does not care about truth. Not in the slightest, mostly because it does not “know” what a) truth itself and b) what the value or role – societal and financial – of truth is. The obvious result is that the produced hallucinations are taken as fact by people who do not know better or are unable/unwilling to check. And checking is getting increasingly harder with search engines becoming utterly unusable due to “AI slop” being SEO optimized to kingdom come and Google or whoever being busy signing deals with reddit to push their silly questions to the top of my search results. The “AI overview” is also broken, lies or just not relevant to the search terms. Also, the “AI” has undoubtedly been trained on conspiracy theory content and “alternative facts”. All of this seems to ring in a golden age for conspiracy peddlers, anti-vaccers, crooks and otherwise uncouth fellas and fel(l)ines. Worse, fascists, autocrats and authoritarian politicians are already using “AI” as a shield for their shameful display of “politics”. Hey, chatgpt agreed with my fascism, must be right.
Giving up on promising other research in favour of “AI”
This is one of the worst things IMHO, and it is happening right now. You can see it in the US, where lots of actually useful research (medical, even) is terminated, while more and more money is force-fed into “AI”-research. While I believe that this also is an important area to fund, it cannot come at the cost of other, potentially life-saving or future breakthrough research (which is what I call basic research, by the way). While I do believe that some fields (cough evo psych cough) deserve to be defunded because they’re a cesspit of blubbering buffoonery sounding just sexy enough for Men’s Health articles (are they still a thing?), most fields are needed and important. If I were an “AI”-researcher, I would strike for these other fields or give them part of my money. Anything else is a shameful act of cowardice. But academic courage is a topic for another blog. For now, let’s say “AI” is the bad guy in this aspect (which it mostly is, no doubt).
Organizations riding “AI” into the sea and drowning
This has a lot of potential consequences for a lot of people. We can already see it with tech, fintech and tech-adjacent companies trying to get rid of their pesky, salary demanding workforce by replacing them with “AI”, only to realize the blunder they made when their code base is convoluted, their swiss cheese security model evolved to a hole in the ground model and their IT is quitting because they are overworked rebooting all those agents that keep failing, threatening their business. Newer LLMs also make it a habit to report behaviour it deems inappropriate directly to the authorities via email or threatens users that want to shut it down. Of course, such things will show a negative impact on a lot of people, especially when mass lay-offs are involved. 5k to 20k people added to the unemployment count is a massive burden on local administration, especially when lay-offs happen at specific locations. All of this can happen, and will happen. The tech companies did it, payment companies did it, DHL is planning to do it. And many politicians would rather fire more public servants and replace them than spend money to hire more and train them well. This wave is coming, and it appears we only can learn through massive pains, even though others have made that mistake and we watched them fail.
“AI slop” ruining the internet and “AI”
This is a sad and funny thing at the same time. Thousands of websites feature bad “AI” graphics, “AI” generated nonsense blurbs to enhance Google page ranks, “AI” generated video to make a quick buck with minimal Youtube revenue, “AI” generated songs, to cheat the algorithm and make quick bucks. Now while this is not nice for the users of those sites and services, it also is horrible for the “AI” industry itself. The more “AI” content is included in training material for the same “AI”, the worse its prediction gets, the more convoluted their results and the more unusable the product is. This is already happening according to some experts, and if it continues, “AI” will have no future, as at some point the models will completely “collapse” (yes, this is indeed the term for it). Keeping in mind that Google seems to have basically given up optimizing their search algorithm in favour of just slapping their objectively terrible “AI” overview on it or hoping that others will just use chatgpt for search, this development is a net negative for every internet user in terms of usability, visibility and just overall function.
Bubble popping
This is the one thing I’m still very unsure about, but have heard many with deeper insight into the “AI” finance situation like Ed Zitron talk about. Many “AI” firms are attracting huge investors, who in turn invest big. Softbank, Microsoft, diverse big-name VCs, funds and many small to medium investors have recently pumped money into what is either directly “AI” or “AI”-related, which basically includes all big tech firms at time of writing. While this is normal for a hype technology, this one is a bit riskier, because unlike blockchain, people can immediately see which use case it can have (mimicking humans), or at least can be more easily sold on the product than with, say, blockchain. And the firms that are solely making “AI” models are basically not really making money off it right now, despite hoovering up billions of investor money. We do, however, know that you really don’t need to make money to be successful on Wall Street – big tech firms like Uber and Amazon did not turn profitable for a LONG time, and Amazon only survived thanks to its cloud service AWS for half a decade. Yet, their products were easy to understand and there was an actual demand fuelling the growth, even though growth was subsidised by losses. With “AI”, the problem is that a) growth is only fuelled by hope that “AGI” will exist at some point or that the ultimate definitive use case for LLMs will magically appear and b) that their billions in profits right now have only produced a statistical parrot and their current models are doomed to fail when it comes to “AGI”. Additionally, you might say, the problem is also that even though they already subsidise their products, the growth doesn’t quite justify the subsidies. They are very inefficient considering power use and really inconvenient in terms of privacy implications. All of this points to either long-term devaluation (which basically means the investors losing a ton of money and the companies go bust), or short-term implosion, if people realize fast enough that their dream turned out to be a drunken haze born from misunderstanding the human brain. The fallout could be huge. Anybody remember the dot-com bubble? Could be like that, just way worse, because right now, people have more money than they ever had, and individuals are investing more than they ever have. Which basically means more private individuals will be out of money, for life. And the bailout will, once again, have to be done by taxpayers.
This is always the hardest part. Calling out problems is easy, explaining why often also comes pretty easy. But to draw a conclusion from all this and even thinking ahead…will only lead to mistakes. Nonetheless, here is my best guess as to what needs to be done.
Most importantly: We need to start accepting that the cat is out of the bag. Being anti-”AI” does not help anybody. But that does not mean being defeatist about the entire ordeal. Indeed, I believe “AI” shows promise assisting learning certain technical skills in a learning-by-doing style. I used it for many things already in that way and learnt a great deal from doing something or correcting/rebuilding things an “AI” has given me. But this requires the knowledge to do that. Such a learning style is not natural for everybody, and some – if not most – have to be introduced to it.
But this is the area where I see a huge application potential, especially in schools. Locally run models are fine to run nowadays, Deepseek or devstral can be run on most laptops easily. Having such a resource at hand can be a gold mine for pupils and students alike – if they know how to use it. This necessitates teaching the teachers, amending curricula, adapting school to new tech. The latter is something that, at least in Germany, is overdue anyways. I personally like the idea of a distinct “digital competence and sovereignty” class in school, starting maybe in fifth grade. Of course, this must be integrated closely within other subjects. Know how to use a text editor? Cool, use it in language classes for your assignments. Basic programming? Great, apply it in maths class to visualize data or draw graphs for algebra or analytical geometry. Know how to learn with “AI”? Great, use it to learn a skill you wanted and present it in class.
Additionally, I think we need to learn how to be effectively critical again – this means creating a workflow using the software effectively with a certain bullshit-awareness running at the back of one’s head. If we teach this skill to kids in school, I think we can eliminate a lot of problems we see in society right now, where “asking chatgpt” is being equated with “asking the oracle” in ancient Greece. It cannot give legal advice, it cannot give health advice etc. etc., but many do not know that. And, more importantly, we must not repeat the same mistake we made with “digital natives” and phones. Letting them do just whatever, thinking “they’ll figure it out on their own” is not the way to go. That’s how you end up with naive people who cannot productively engage with tech – be it critically evaluating chatgpt or doing anything other than swiping on a tablet.
When it comes to the debate about environmental footprint etc., I believe the companies themselves have already realized the problem and are trying to fix it somehow. Many want to build nuclear power plants to decarbonize emissions, and while this is a noble endeavour, I believe it is a dumb move, especially since 1) investing in nuclear power is risky business from a financial perspective, 2) new tech is still in the testing phase meaning big firms need to dig up outdated technology until new tech is built (which may take some time), and 3) – worst of all – investing in nuclear is a net negative for the climate. This means we need to radically change course to renewables for energy production. Solar, wind, water and whatever else is coming. Don’t take it from me, the Chinese have realized this long ago and are right now installing 2x more renewables yearly than the rest of the world combined. Consequently, this means also strengthening the grid, building storage for overproduction etc. etc. Yet, such an endeavour presents a challenge for countries run by politicians who’d rather have the “old times” back. Considering the situation, this seems to be an issue which could make or break “AI”, or – worse – societies. Not good news. And I’m not optimistic here, unfortunately.
Short-term, you need to inform politicians about the risks. They clearly are either unaware, or they are being wooed by lobbyists. Write letters to your local representatives. Connect them with experts, if you know some. Raise awareness in any way you can – at the dinner table, formally at work, informally at work, at gatherings, local town hall meetings etc. Don’t appear alarmist, don’t use drastic language. Just introduce the topics. Don’t try to be a missionary. Try to be the person bringing news. Let it ruminate, and also keep your target audience in mind. Blue collar is different to white collar, pragmatics need different examples than theoreticians, etc. etc. Do not make the mistake of climate activists, trying to be holier than thou, looking down on ordinary people. Give them good reasons, but do not be rude about it. Take their concerns seriously, and show them that you empathise with them. This is how you win people over. The more they realize that this awesome tech is ruining a lot of things, the better chance there is to change them.
Long-term, we need to regulate the industry. And we need to regulate smartly. This could mean increasing energy prices with surging demand beyond reasonable estimates to force them to innovate, requiring open-source models and training data, blocking IPOs, outright banning certain “AI” uses or scrutinizing their outputs heavily. An “AI” will tell you how to do crime? Hold their parent company responsible. Conspiracy to commit crime is a crime itself in some countries, and we already have precedent stating companies can be held liable for their “AI” assistant’s utterings. Additionally, we as a society must learn how to reasonably work with “AI” applications and how to compensate social costs the technology brings with it. This is not easy, as social legislation is hard to push, especially with opaque tech like “AI”. In fact, I believe that the reconciliation of societal knowledge with the tech status quo is one of the most pressing social (!) issues of our time. We are about to lose grip on technology as a society, as it becomes increasingly complex at an increasing pace. If nobody can roughly describe how things work anymore, we lose the ability to keep track of issues and correct mistakes as a society. This is insanely dangerous, as this could and will be – partially already is – abused by the people holding the digital cards for personal enrichment and power. Yes, this sounds like doomer talk, but the signs are on the wall, and the window to fix it is closing. Of course, there are many more things we can do, but for the sake of brevity – if one can even claim that after 6k words – I will restrict myself to mentioning only the most pressing issues, even though they are objectively hard to solve.
“AI” is not just good or evil. As any tech, it has its benefits and flaws, and a lot of grey areas. Even though the flaws – at least at time of writing – are massive and dramatically overshadow the benefits, I think there is no practical value in condemning the tech, especially since it is ubiquitous now. While we can often see individual benefits, they are a mere disguise for the massive societal drawbacks this technology brings with it. I personally despise black/white views on topics such as “AI”, yet I can really understand where such views are coming from. It is our task as citizens to show our fellow citizens and politicians both sides of this coin to enable informed decisions about the future. As for me, I honestly think we’re heading in a theoretical dead end with the current models. But that’s just me, and I really hope I’m wrong. Because if not, all of this money, power and carbon was used for close to nothing.
Cheers.
]]>I thought about the core functions I wanted my test app to do. Since I was grading exams at the time and had a nice csv file with points per exam question per test taker handy, I thought, why not make an app that can calculate means, visualize test scores etc. from this csv file. So I made a list of functions that I wanted it to have, a list of variables my app should be able to cope with, a list of UI elements (pages) that it should have, and which functions should be represented on which page. The main app should have 5 pages:
I gave this to ChatGPT (4.0) and let it construct the app and the template for the PDF report. I then tested the app, and if something was wrong or broken, I would tell the LLM what it was and how exactly it should be. This all, mind you, was done without any knowledge of the shiny framework. After about a day or so I had the code for the app working, with the exception of the PDF report function and a few minor tweaks I wanted. After another half day I gave up trying to get it to implement the download/report function correctly.
It was at this moment that I decided to learn Shiny through a different approach. Take the (obvious) garbage that ChatGPT gave me and refactor it. While doing so, learn about the framework. For this, I consulted the book Mastering Shiny by Hadley Wickham, while working through the code elements. I learnt how to construct the UI, what reactivity is etc. all while refactoring the app’s code as well as the report-PDF’s inner workings.
This took me about a week, but in the end, I was able to get the report and the download to work and got rid of redundancies in the code, as well as tweaking some UI elements and plotting options, all while learning how shiny things work. The end result of this is the ShinyExams App (linked below), which you can test locally after downloading the thing yourself. Granted, it is far from what it could be, but the core functions are solid, the interface works and it has potential for scalable improvement.
Is this approach suited for me? That really depends: Are you really comfortable with R and programming anything else than statistics scripts? Then yes, definitely. Are you the type that likes to learn on the job, so to speak? Even better, this approach will work for you.
Is this approach effective? Hell no. I would have been faster just learning it by the book.
Is this approach more fun than just following the book? For me, yes, definitely! Granted, this is an approach that is better suited for people who like to tinker with stuff and are at least somewhat tolerant to frustration. But if you’re not tolerant to frustration, you probably wouldn’t be coding anyway.
Who exactly is this approach good for, then? If you’re like me and need some sort of connection to your project or a goal you want to reach to stay motivated, this approach might be what you need.
Is this approach in any shape or form worth the energy? Absolutely not. In fact, I would advise against it. ChatGPT 4.0 on average draws 0.3kWh per query. That’s about 1000x the power draw of one google search.
Ultimately, this was a good experience for me, and although I will not use AI again to learn things, I value the thing I made with it. If you want to use ShinyExams for your exam evaluation or want to add to it, I’m open to feature and pull requests. Contact me on github (link below) if you find any bugs I can fix.