Working on a new text editor…

So, as I’ve been teasing (for like 2 years at this point…) I am working on a text editor. Originally it was going to be based in the terminal, and using the Aquila virtual machine, then using Blæst, finally using Crow and not in the terminal. So here are some things I learned.

Keep it simple stupid

First of all, make everything easy on yourself if possible. It’s not always a good idea to reinvent the wheel. Case in point is the text editor interface itself. For so long, I was hung up on how the interface should work. Usually I figure this stuff out and start work, but for the obvious reason of “hey a gui library is pretty freakin huge” I really wasn’t able to get any work done on the project. Especially too since the more I stalled it, the more ideas I got. So eventually it turned from “let’s have it in the terminal” to “let’s have a library to create guis that seamlessly convert from the terminal to literally any ui toolkit and seamlessly integrate into the system they are running on. Needless to say, eventually it just became “what if the text editor was actually an operating system”, and magically that’s where I stopped. So for one, know your limits and keep within them. Now the editor is being written in Qt 6, mostly just because Qt in my experience has the best cross platform ability, and it is really easy to theme from within an application itself. But maybe Qt wasn’t the best option…

Do NOT use Qt

So this might or might not come as a surprise, but I really hate Qt. Always have and always will. The main reason I dislike Qt just mostly has to do with it’s weird licensing problems. Like “is Qt really free software”? If questions like that have to be asked, you probably shouldn’t be using the software in free software projects. Pretty big coming from a guy who uses Mac OS now, but hey, software freedom is something I still really care about, and I want my text editor to be enjoyed and used by all, so picking a UI that was FOSS was an important thing for me. So what do I mean exactly for “is Qt really free”, well for one, now you have to download their (probably proprietary) Qt installer, which requires that you make a Qt account. If you want to have an offline installer, or god forbid just download the headers and libraries, you need to pay for enterprise to do so. So you have two options: embrace freedom by proprietary software, or empty your wallet. If you can, build Qt yourself, or get builds from your distribution or operating system, I just needed mainline Qt so I could make sure it works with what most people will probably try building with… Big mistake.

Make sure things you say are “embeddable” are actually embeddable

Is it a coincidence that a whole boat load of changes came to crow the minute I picked the text editor job back up? Crow got a whole lot of things, including CMake support (hmmm… doesn’t Qt use CMake…) and in general just refining the structure of how things are stored and accessed to allow for crow to be built much easier into a shared or static library for use in other platforms. Eventually when I start making AIs and robots in crow (when I finally make the move away from the dreaded JavaScript abomination), you’ll probably see a multitude of fixes and changes in those as well. Funny how that works.

Just don’t make a text editor in 2022

Now this one may seem anticlimactic, but just don’t do it. Text editors are a dime a dozen now, and chances are you can pick up and maintain one that is exactly what you’d like and maybe more with less time commitment than doing it from scratch. Now I say this with compilers, that there is a great deal to learn from making a text editor, but to be honest, you can create far more useful applications, such as a rich text viewer or chat client, and use the same skills a text editor would use, and maybe even expand on them. Personally I have hated writing a text editor, but I’m doing it for completion at this point. I’ve said I’m going to do this for years now, and I’m damn well going to do it.

Posted in Programming | Tagged , , | Leave a comment

Just moved to Boston

If you’re wondering why there have been no new posts lately, I just moved to Boston. Still getting a lot of stuff setup here, and had to leave a lot of my development hardware behind, but hey, everything is progressing well with Crow (formerly colang). A release candidate for release 1 should be coming within this month.

Anyway, be on the lookout for more posts soon. Side note: WordPress has a surprisingly nice mobile app, I wrote this entire blog post from my phone while waiting for the T, pretty sick.

Posted in Meta Posts | Leave a comment

Switching to Apple

So, for pretty much all my life I’ve used Windows computers, and ever since 2016 pretty much used Linux full time. I used Linux as far back as 2011, with Ubuntu 11.10 being my first distro, however 2016 was when I truly hit my Linux stride. Since then, the standard UNIX-y way of doing things has become permanently ingrained in how I work. Before switching to Linux, I used Visual Studio and Code::Blocks, and after I primarily used the command line. I also switched to using debuggers like gdb which I found much easier to script and run batch tests, and also switched to using programs like git for version control, which was always really clunky to use on Windows. Basically, using Linux taught me that Windows is a terrible development environment for basically anyone, and especially if you intend on making cross platform applications.

Fast forward to now. I’ve been using Linux in some capacity for over a decade at this point, and I still really like it. Coming from Ubuntu 11.10, we were still using Sys-V Init back then, can you believe that. And if Wi-Fi worked out of the box, you had basically just won the lottery jackpot. I still very much like Linux, and like the direction Linux is going. But with all of that being said, I decided that the time was right to try out Apple and their MacOS. Soon, I am moving to a new apartment, and one problem with that is I will be forced to severely downsize what I own. One unfortunate reality is I can’t bring my massive computer setup with me, it is just simply too large. I really needed to get a laptop, and while I was looking at buying a thinkpad (like I already own and use for school), I decided to spend the extra $1000 and buy an M1 pro Macbook Pro 14 inch.

My first impressions of the computer were very good. The screen particularly blew me away. Usually I tend to be the kind of person who spouts things like “Screen doesn’t matter” and “If I want a high resolution screen I’d buy a TV”, however, I work a lot with CJK Language, or Chinese, Japanese, and Korean. CJK languages use very fine symbols, which when rendered at lower font resolutions makes them pretty much unreadable, or at the very least heavily pixelated. Thanks to Apple somehow cramming something like a 2k display into a 14 inch laptop, CJK fonts are incredibly clear, and I can already feel myself never being able to go back to even just a 1080p screen for CJK. Along side the screen, everything felt really snappy. Animations lasted just long enough to be known, but not too long to feel sluggish. Everything seemed to run pretty fast. Overall the thing I was most surprised by was the lack of screen tearing and other Linux-y things I come to expect with the UNIX-y base. I unfortunately had to switch from GNOME/Wayland back to MATE/X.org recently as Debian broke the sid GNOME packages. Wayland was really nice, but ever since the breakage, I have been struggling to get MATE to run without screen tearing without making the entire desktop lag horribly (thanks picom). Overall though, from a visual perspective, MacOS blew me away.

Another unexpected win from MacOS was customization. I had pretty much always been told (by people who don’t themselves own Macs or use MacOS) that Apple locks everything down, so you experience the operating system the way they intended. And while this is true of things like the overall application theme, or the general feel of the desktop and OS, this is not the case for the applications included. I was very angry that Finder kept opening the “Recents” menu when I would open it. To open the Home folder, I would have to navigate to the menu and physically click “Go > Home Folder”, which was a massive pain, especially when I would then have to navigate into further folders from my Home folder. I thought this was going to be the moment I became one of those cynical Apple users where I convince myself that my purchase was justified and “I just have to live with these few extra steps”. Didn’t help as well that I could not for the life of me find any way to set Finder to default to the Home folder anywhere on DuckDuckGo, so I thought I was completely out of luck. However after just deciding to check the preferences page, I found not only could I set Finder to open to the Home folder, but I could add the Home folder to the side bar, and even add the hard drive shortcut to the desktop like in OSX of old. To be honest, I had been trained by general consensus that Apple preferences do basically nothing, but to my surprise, I found everything I had wanted. Considering you cant even set this on GNOME (at least without touching dconf), that’s incredible. Along with this as well, I was pleasantly surprised to find I could exclude the terminal from all the security checks that MacOS does to applications running. Normally this would be totally advised against, since it completely undermines security, but for the terminal, I am mostly only running applications I built from source, and thusly audited, so doing this stopped the annoying “Do you trust this application” popups which I also saw non Mac users complaining about.

So, if you have had no experience with Apple, I would say go out to the nearest Apple store and give it a try. Or you can emulate it on Linux using OSX-KVM if you don’t want to go out or buy anything. I was very surprised with how good Mac OS was in my experience, as well as Apple hardware. Does it justify the price? That’s up to everyone to decide. However I am very happy with my purchase at least at this time. I will make sure to update if anything changes with my opinion on it. Also just a side not, Apple gives some pretty great discounts for students, and they can’t really even prove that you are a student, so if you look young you can probably pull one off.

Anyway, I also bought an iPhone SE 2020 recently as well, but that’s an entirely different story for another day. I’ll probably write another blog post about it (once I’m done with my super massive blog project I’ve been working on for a week at this point). So stay tuned for that.

Posted in Programming | Tagged , | Leave a comment

Back to MATE

So… Debian pushed out a broken update to GNOME, so now I’m back to using MATE. And likewise, I’m back to using xorg. Being honest, I haven’t had such a bad time switching back, I may end up staying with MATE for a little while now. I have always like GNOME 2, so MATE isn’t a desktop I hate, but I do like the looks of the newer GNOME as well, particularly with libadwaita. So… welcome to Linux I guess, but hey, MATE isn’t a terrible fate to be suck with. To think I could be stuck using i3 for a while… horrifying…

Posted in Linux | Tagged | Leave a comment

Meta Post: I enabled comments

Just a quick post, I found out hcaptcha is available for wordpress, so I decided to install it and re-enable comments. I had them disabled because I had problems with spam way in the past, but now hopefully that shouldn’t be a problem. If spam picks up I will disable them again.

Posted in Meta Posts | Leave a comment

80% of the problems, caused by 20% of the bugs

If you’ve been around the internet for a long enough time, you’ve probably heard of the 80-20 principle, or Pareto Principle. For some reason, it seems like everything is linked to the proportion 80 to 20. For example, 80% of the world’s wealth is held by 20% of the population, or in my case, 80% of the problems were caused by 20% of the bugs, even if not obviously caused by them. While I was skeptical about the claim before starting on my most recent project, now I am a full on believer.

Recently, I’ve been writing my own dialect of Lisp. I mostly am doing this as an exercise in creating high(er) level languages for embedded systems, cutting a lot of the fat that current high level languages like Python have. And I chose Lisp because Lisp is cool. While all of this is a blog post for another day, what to take away from this was that I decided to write this for every system ever made, meaning I had to choose a very common programming language to base on (especially more important since Lisp is interpreted, not compiled), so I chose ANSI 89 C.

When I started writing my Lisp interpreter, I was doing so very fast. In C, you are the memory management, meaning if you don’t do everything correctly, your program could start executing undefined code, or return undefined values. While starting, the most bugs I had were uninitialized values for variables. For example, in the following code, we have no idea what the value of x would be. It could be 0, it might not be. Its really up to the compiler to decide if it should be initialized, and up to where on the stack it is allocated to say if it already has a value.

int x;

printf("x is %d\n", x);

Now, these values are easy enough to find, simply make sure to pass the warning flag -Wuninitialized to the GNU C Compiler (or Clang as well), and boom, every uninitialized variable hopefully should be highlighted. This worked for me, so it was overall an easy fix. However as I continued on with the project, I ran into stranger errors dealing with memory management which seemed by all odds to be coming from a billion different directions.

Now, I have been programming in C/C++ for a little under 10 years at this point, having started sometime around 2014, so I know my way around. But I also have a tendency to be very lazy when programming, especially for proof of concept things, which usually leads to a “I’ll finish this up later, its not like this is production code or anything” type of mentality when working. Usually what this means in practice is neglecting memory management, since “you can just allocate a buffer of like 1KB and it should never overflow… right?”

Working on my Lisp interpreter, I first ran into problems of programs that were too large having variables overwritten randomly. In the interpreter, variables just hold a direct value, usually a pointer to memory, or an integer or decimal value. If it is a pointer to memory, it adds it to another list, holding all active memory allocations. This list of allocations allows the garbage collector to determine whether or not memory is in use, and frees it if it no longer is. This is a pretty nice system, if only C had it. I was quickly having problems of allocations being overwritten with seemingly garbage data, which caused memory that was still in use to be freed, which would cause undefined activity when it was called upon in the future. This is particularly a big problem on Microsoft Windows, which has a very strict memory model, which allows for zero out of bounds memory reads. Basically, on Windows, if you free it, its gone forever. Now, this would usually cause a crash on Windows, but not on Linux, so for a while, I just assumed the Windows version was completely broken because “Windows”.

Next problem with memory management was the freeing of garbage data. For example, if the variable list overflowed into the allocation list, then garbage data could be written to the memory stored as an allocation, which when no longer needed, could cause the system to try to free memory that it has no right to free. And likewise if the allocation list overflowed, it would spill into the openFiles list, which contained references to open files being read by the interpreter. Again, a file could be closed that would result in undefined behavior, especially if the file descriptor wasn’t open. This happened while porting the interpreter to classic Mac OS version 6. And again, I chalked it up to “man Apple really sucks”.

These bugs really took a toll on me, especially because I was watching them in debuggers, and just magically seeing values overwrite other values, for seemingly no reason. It was especially strange too, since when the program would eventually crash for what it did, the crash point would be usually a strange location, somewhere very far from where the problem was taking place. And by this point, the memory management aspect of the program had long since left my current focus, as I just assumed “Its worked this far, it should keep working”. So I wasn’t even really considering that bad memory management could cause all of these errors, which led to multiple long nights of adding unneeded sanity checks, and attempted error correction. Remember kids, fix a problem, don’t try to patch around it.

Eventually I was running out of options to try to fix things, it seemed like at this point, every internal function had some crashing bug. So I decided to run the program through valgrind, a memory debugger among other things, and man, I should have done that much earlier. To my absolute horror, I was making overflow write after overflow write. I wish the GNU Debugger had been more up front with me about that. Once I fixed all the overflow writes by keeping track of buffer size, and reallocating when needed, magically all of the bugs went away. And so too could all the shoddy patchwork. Overall just fixing about 3 buffer overflows made the program about 1000% more stable.

Now, I’m not going to act like my Lisp interpreter is anywhere close to being done (although stay tuned if you’re curious, I’ll probably make a few more posts here about it in its lead up to version 1.0), but it sure as hell is a lot more stable. So in closing, somehow, I guess the 80-20 principle is true. If you are having a problem with your program, or a program you are using, look for the most obvious sources of error, because in all likelihood the obscure bugs you have to work for to get to take effect probably aren’t the ones that are causing your program to crash, its probably the ones like I’m facing, even if it doesn’t seem like it.

Posted in Programming | Tagged , , , , | Leave a comment

The sad state of input methods on modern Linux

こんにちは、私はコリンです。よろしくおねがいします。

So… I decided this blog will also be a place for me to post rants. So lets start. I speak 4 languages: English, Spanish, Mandarin Chinese, and Japanese. 2 of those 4 use a non Latin character set, so I have to use input methods to type characters in those language. This is not just relegated to far eastern languages, but ones like Russian and Arabic too. Input methods have long been the way to type in these languages, mostly because it solves the problem of “how do we translate this language into a set of keys to easily type on a keyboard.”

A language like Chinese is ideographic, which means it was never meant to be typed, but drawn. While a few Chinese keyboards do exist for typing by “creating” characters from strokes printed on the keys, its much easier to learn Pinyin, the Romanization of the pronunciation, and learn to type that way. However if all chinese was typed that way, we would get sentences like “wo shi xue sheng” rather than the actual Chinese characters “我是学生.” Not to mention, since chinese has many homophones, like “是 – shi – to be” and “十 – shi – the number 10” however the first is pronounced as “shure”, and the second as “shee” thanks to the tones attached to them. (In Pinyin, there are marks to denote the tones, but to type those you ALSO need an input method.) Thankfully input methods exist to take the Pinyin you type into account and turn it into characters that fit into context.

Now all of this sounds like a net positive, and for the most part it is. Both Windows and MacOS have built in support for input methods which are easily accessible from the settings, and Google implements a really damn good one into all Google web apps. But Linux… well… they kinda work on Linux.

In all fairness, I have used input methods on Linux for the better part of almost 5 years now. Originally just to convert my QWERTY keyboard into Dvorak (another good use for input methods), but more or less now to convert typed Chinese and Japanese into their correct written form. For the most part, both major input methods, FCITX and IBus have worked alright, however now we are in the transition between GTK 3 and GTK 4, along side the transition from X11 to Wayland. This transition has truly thrown a spanner into the works for the input methods, and just goes to show what Linux needs is more of a stable backbone instead of the module fluid system that exist now.

In Linux, both IBus and FCITX seem to hook into the application’s text input, rather than the core system’s text input layer. This is good as it means the core daemon doesn’t need to be run with root, but also means its up to each application to honor input methods. On X11 this is much easier, because all you need to do is have X11 honor the input method, while this doesn’t guarantee every application will still work with it, there are still less moving parts. On Wayland however, the problem gets worse. Wayland expects every application to be in charge of everything itself. Everything from drawing its titlebar and decorations, to honoring input methods. Personally I have been using Wayland for the better half of a year now on GNOME, and I haven’t had as many issues with input methods arising just from Wayland as I thought I’d have, but some applications, particularly ones based on alternate GUI toolkits (not GTK or QT) seem to have the worst problems. With those its just more effective to run them in XWayland and hope that works with the input method.

Although this isn’t to say X11 doesn’t have its own issues with input methods. For some reason, Wayland or not, it seems like GTK and QT universally disrespect the input method. I’m not entirely too sure how the input methods work low level, so I’m not exactly sure how they send their data, but what I do know is multiple environment variables are needed to make sure you get the input method working in applications. Usually this means adding a few lines to your `.bashrc` or what not. Here is my config:

# Can't believe I have to do this in 2019 + ?
export GTK_IM_MODULE=ibus
export XMODIFIERS=@im=ibus
export QT_IM_MODULE=ibus

So all of this might be a little alarming, but for the most part, somehow this horrible beast of input methods just seem to work fine. The problem is when technology changes. With the switch from GTK3 to GTK4, I noticed a lot of applications that traditionally worked with input methods no longer work. Whether this is a problem with IBus not supporting GTK4 (despite being integrated directly into GNOME), or GTK4 speaking a different language to the input method, or hell, maybe I need some new undocumented environment variable. This mostly came about when using chromium and other chromium based browsers, which I discovered an override for thankfully.

GTK_IM_MODULE=ibus vivaldi --enable-features=UseOzonePlatform --ozone-platform=wayland --gtk-version=4

Now if that’s all I needed, I would be happy, however of course something has to go wrong. Thanks to flatpak being a sandbox (and flatpak is where I have chromium downloaded), it doesn’t like to play nice with IBus. I’m not too well versed in flatpak, so for all I know, maybe I need to install some module to make it work, but right now EVERY flatpak app cannot work with an input method, like at all.

To be honest, all of this should not be necessary if Linux ever wants to be a household name, especially for people in countries where they don’t use a standard Latin alphabet, or people who speak multiple languages, but want one keyboard. The amount of time I have invested into troubleshooting input methods is insane, and in all honesty, pushes me more towards Windows or MacOS for work in foreign languages, because at least those platforms “Just work :tm:”. This was made worse when during my tenure at a Japanese university, I used only Windows 10, and the experience was so good that all my work in Japanese now is done on Windows.

Hopefully this rant post isn’t too annoying or meandering to find some meaning out of. If you want more information on how to setup an input method, either to torture yourself or otherwise, this arch wiki article provides pretty good information on how to do it. In all honesty, input methods on Linux work. Like period, they work. Its just the sometimes that they don’t, they really stand out more than the many times they do work, especially when on systems like Windows or MacOS, they work constantly. I’m not really too sure how to go about fixing this that isn’t simply “run the daemon at a system level” or “make every Linux application use the same GUI libraries”, all of which the Linux community would hate. So basically, we just have to live with it. いじょうです。

Posted in Linux | Tagged , | Leave a comment

So I finally got around to setting up a blog…

I’m going to be honest, I used to really make fun of wordpress, but its really easy to setup. Hopefully this is easier than my horribly janky system that used Blæst as a markdown parser that was terrible and barely worked. Particularly it was awkward that the æ in Blæst would cause the ENTIRE program to break, so it had to manually go back and edit it in. Hopefully wordpress is easier. Also too my friend Brandon already set up his blog on wordpress and it looks pretty nice so…


Here are some things I really like about wordpress so far

  • The themes are a nice hit of nostalgia. I remember the web before sites like WIX and Squarespace took over, and especially before frameworks like React took over. What’s nice about wordpress was that it just seemed to slot in perfectly with my already existing (and rather bare bones) web server. That’s pretty cool in my opinion.
  • WordPress seems to organize everything really nicely. Being able to write posts and categorize them from a nice panel is pretty convenient, especially since before I had to do it all manually using markdown.

Now the obligatory “How I setup wordpress” section

Setting up wordpress was honestly almost too easy. To start, I use Debian stable with the apache web server. I do NOT like to add external repositories, so I do everything manually (this is assuming someone has packaged wordpress for Debian already). I also do not have the ability to run Docker or other container programs because I lack disk space. So I basically am used to having to compile and backport my own programs, and sometimes go through hell doing it. But wordpress was different, it just seemed to work directly out of the box on Debian stable, so here’s how I did it.

Assuming you use apache2, the installation should look something like this.

First install the needed packages.

 $ apt install php7.4 php7.4-mysql libapache2-mod-php7.4

Next, we need to create the folder in your web directory for wordpress to live. For this example I will use https://website.com/blog, which would be located at /var/www/html/blog

$ mkdir /var/www/html/blog
$ cd /var/www/html/blog

Now we need to add the wordpress database and user to our database server… Oh… and install a database server

For the database server I decided to use mariadb, seems like everyone uses it.

$ apt install mariadb-server

Now once you have mariadb installed, lets add the wordpress user

$ mysql -u root

From here you are in the mariadb interactive shell, so we need to type SQL commands.

Let’s create the wordpress database:

CREATE DATABASE wordpress;

And the wordpress user:

CREATE USER 'wpuser'@'localhost' IDENTIFIED BY 'securePasswordHere';
GRANT ALL PRIVILEGES ON wordpress.* TO 'wpuser'@'localhost';
FLUSH PRIVILEGES;

Make sure to update the username and password shown above if you feel like it. Also make sure to remember all of this for when we setup wordpress itself.

Now leave mariadb by just typing:

QUIT;

Okay… Now we are prepared to get wordpress setup.

First restart apache2 so our modules update, to do this just run

service apache2 restart

To download wordpress, enter your directory where you wish to install it (we are using /var/www/html/blog) and type:

wget http://wordpress.org/latest.tar.gz
tar xf latest.tar.gz

This will extract to a folder named wordpress/, if you don’t want that, simply run

mv wordpress/* .

Now make sure to allow wordpress to write to the folder, for some reason, if you don’t do this, wordpress will ask for FTP credentials to log into your server (why…?).

chown -R www-data:www-data /var/www/html/blog

Now navigate to the site and directory you installed to, and just follow the few instructions, and boom! Now you have wordpress!

Posted in Server | Tagged , | Leave a comment