Arrow of time
Arrow of time

Archives

How beautiful the PostgreSQL planner can be

Just wanted to share how beautiful the PostgreSQL query execution planner can be. Here is a query where I want to find out both the minimum and the maximum of the id field, which is a sequential primary key: explorer2=# explain analyze select min(id), max(id) from expl_iocombined;...

Supercapacitor experiments #1: just a LED

When I was a young student (in a third world country, mind you), around Y2K, we learned about capacitors in pF, nF and uF ranges, that's pico-, nano- and micro- Farads, i.e. 0.000000000001 F, 0.000000001 F and 0.000001 F. All these are useful and used in practically all existing electronics. They are also very small amounts of Farads. Consequently, one whole Farad was considered ginormous, an almost mythically high capacity. I remember one student...

Introduction to Go for Python / Django developers

This is a gentle introduction to Go for primarily meant for Django developers. Admittedly, it is in some ways like comparing apples and oranges. Since Go is a language (with a batteries-included standard library) and Django is a web application framework, this guide will make some choices on how to make Go useful for web application development. There is a very large number of Go libraries, frameworks and micro-frameworks out there, and more are developed...

C++ Atomic Types / Memory Barrier Performance (or: do we need CPU caches?)

A friend of mine, a computer scientist working on memory consistency models theory in C and C++, has relayed to me an extraordinary claim, which boils down to this: "using memory barriers in code for each read and write does not have an impact on performance." Now, this claim is something he's heard second hand, and since he is an extreme example of a theory-oriented guy, the validity of this claim was not something he...

TIL: How do bicycles (and cars) turn?

It occured to me while driving a car that the act of turning direction is actually non-trivial. Reducing the case to a bicycle, consider this: You are pedaling your bike, and driving in the forward direction (of course). You now have certain speed and acceleration vectors pointing in front of you. You stop pedaling, and decide to turn left, 90 degrees to your current direction. You turn the wheel. Depending on the angle of the wheel and...

Bitcoin works exactly as intended, get over it

There have been some alarmist articles recently which basically boil down to "Bitcoin is doomed! Abandon ship!", and while I agree with some of the points given, I think that almost all of them, together with most of the current users of Bitcoin, are a bit missing the point of its existence. Here's my take on it. Blockchain itself is independent from the networking part, and it works fine The blockchain is basically a way...

Why Oculus Rift is not the future of VR, and Cardboard is?

Because of availability. Thank you for reading, and good night, see you again sometimes, don't be a stranger! Explanation I didn't think it needed an explanation util I talked with a collegue of mine who almost completely disagreed. Here are four reasons why I think that, unless corporate shenanigans fuck things up, a descendant of Google Cardboard will still exist in 5 years, while currently hyped darlings like Oculus Rift, Samsung Gear and others will be forgotten. #1: VR...

Using Go for web app development

Last year I've been considering and experimenting with learning another compiled programming language, which would supplement Python which I normally use for, well... almost everything nowadays. There's really only one objective reason for this: performance. With Python, it simply never gets better, and Python 3 is a mess, with basically non-existent PyPy support. Sure, I could write performance-sensitive parts in C (or C++) and use them from Python, but... no, just no. The things I...

Developing Bitcoin services - general notes

For some time now I've been developing back-end stuff with Bitcoin, and here are some things I wish I could have learned the easy way: If analysing the blockchain (for whatever purposes), it's all fine and breezy until about block 200,000 - that's where the fun starts. Don't test your code with earlier blocks, since compared to the more recent ones they are trivially small. Python is fast enough, PyPy is faster. If you're using bitcoind RPC,...

Story: Court of the Red King

1946. On an almost black and white day in November, the plaza was full of solemn people. Tens of thousands of them, in fact, had gathered under the gray sky, in the semi-circular theater surrounding the huge podium, in the middle of the busy construction project of the Great Hall. It was said that, once completed, the monster building would take two hundred thousand people, all watching and enjoying the power of their Reich....

How are successful open source projects started and what keeps them going?

I've been thinking about what drives the core developers of successful open source projects, or in other words, what keeps the projects going. These are just my own thoughts, they may or may not be correct or applicable for any purpose, YMMV. Being done by human beings, open source projects depend on those people being motivated. In general, people are motivated by similar things, and those, I think, are in open source projects...

Go: methods, receivers and benchmarking

One interesting feature of Go is how easy it is to write tests and microbenchmarks for it. In fact, tests and microbenchmarks roughly folow the same syntax. Another interesting "feature" of Go is how it introduced subtleties in language which can be tarpits for beginners and those familiar with other languages. Take for example the Go equivalent of classes. There are no classical classes here, but a way of doing things which is similar to...

Trying out Nim, D, Go and Rust - part 1

Currently, if I want to describe myself as a developer, I say I'm a "C and Python guy." What I mean by that is that I like to work best with C and Python, and seek such opportunities - but it didn't start like that. I basically grew up on BASIC and Pascal and counting by LOC I've probably written the most in PHP. I also have experience with C++, assembly, Java, C# and JavaScript,...

Why do you need an Encrypted Notepad?

If you are like me, you probably have dozens (or even hundreds) of passwords for various services and systems which you either don't want to "remember" in your web browser, or the services are not (shockingly, I know!) browser-accessible. You have probably already remembered to search for an application for your mobile phone, or your desktop, a laptop, or whatever, which would to the job of remembering "7i48!fac3" for you. And then, like me, you've realised that there...

Why FreeBSD's pkg sucks

A not too long time ago I was a big FreeBSD user, with dozens of production installs. Gradually, I've been using it less and less and now I feel I must describe why, in a hopefully productive and positive fashion. In short: it's all about the packages and ports. An operating system is useless without its applications, and the currently blessed binary package management system, the pkg is seriously broken. I was very enthusiastic about pkg (then called

Installing Django on Windows

Though it is obviously a blasphemy, sometimes it is actually necessary to run Django on Windows - mostly due to a client having based its infrastructure around the Windows ecosystem. The "blasphemy" part comes from Django being a really nice framework which evolved on and for a Unix-like system, and it relies heavily on features like WSGI, FastCGI, and command-line tooling which are foreign to the Windows environment. Luckily, the compatibility is improving, by features...

Online.net physical ARM-powered cloud

Online.net is a French hosting company offering the usual range of services from simple web hosting for 1 € / mo to dedicated servers, but with a twist I haven't seen before: their offer of dedicated servers is incredibly affordable! See this example: for 6 € / mo you can get yourself a dedicated VIA Nano x86 CPU powered server completely for your own usage. These CPU's performance is comparable Intel Atom vs Xeon performance

I recently got myself a small laptop, an Asus Transformer Book T100TA, and I'm surprisingly happy with it. The current generation of these devices is what netbooks were supposed to be: very portable, with a very long battery life, but still powerful enough to be usable. In fact, I'm writing this blog post on it. Finally, both the hardware and the software (the device is running Windows 8.1 and I'd be crazy...

On (artificial) intelligence

Consider this armchair philosophising on the possibility of artificial intelligences. My stance on the existence of our biological intelligence is that it's a matter of accident, in the sense that all evolved traits are basically accidents: random mutations which get selected for over a large time span. It was very probably honed during that time, as individuals which had "more of it" had an advantage over the others. However, something singular must have happened some time ago, some...

Story: The Sin

It looks like "the writing bug" comes with the territory of being a nerd with an abundance of ideas and a complete lack of time to do them in. I have been writing mostly technical articles for almost 10 years now. In this time I've written literally hundreds of articles, at a rate of approximately one every week or so, most of them for a real, printed IT magazine in Croatia named Mreža. That's all in...

Why I listen to Nick Cave

Just listen to this as the sountrack to this post. I hope it will not be a long one. A few days ago I was surprised, and a bit amused when some of my friends told me they think I'm a pretty cheerful and relaxed person. Which got me thinking of the ways in which they were right and where they were not. Soon after that event I've watched Nick Cave's...

We should replace asm.js with Rust

I can't be the only one to think there is something horrible about today's JavaScript. It is like a Lovecraftian programming language from outer space - it is here to stay, and if you follow certain forums it looks like it's spread absolutely everywhere and absorbed every other language, but at the same time it just seems so damned indescribably icky. It was a toy language that went nuclear. With just...

The Needle Search Server - alpha

I've written before about my Needle light-weight full-text search server. To recap: it's a full-text search server written in C++ with a FastCGI interface, using Google's LevelDB for storage, and with a pure REST API. It's available at BitBucket if you want to test it yourself! As these things go, it took me a lot more effort to find the time to work on Needle, but I'm managing it here and there....

What Django caching can do

Nothing special here, just wanted to make a note about how simply turning on Django caching can bring a site's performance up from this: Server Software: nginx/1.2.1 Server Hostname: vo.ivoras.net Server Port: 80 Document Path: /api/location/1 Document Length:...

My Font Awesome to bitmap converter

A few months ago I created a small script which extracts Font Awesome glyphs and creates transparent PNG images, with some fancy additional options such as shadow / emboss. My friend Saša then prettied it up a bit and we basically forgot about it - I didn't even describe it on this blog :) I've just needed an image from this project so I'd thought I'd finally write...

Amazon's E-book Price-slashing Campaign and on Replacing One Evil with Another

Earlier today I have received a letter from Amazon, which I'm going to copy-paste below for completness and archival. It basically tries to invoke sympathy for Amazon in its "debate" with a publisher called Hachette. I don't follow the publishing world that closely and I admit to never hearing about Hachette until earlier today, but I am very aware of the problem at hand, which is pricing and distribution of digital goods...

How do you explain an OS kernel to a layperson? + an old text about The Matrix as an Operating System

Earlier this year I was a guest at a gathering of people who were mostly involved with social sciences and politics and a topic soon arised in which I had to explain some of the things I work with. In this group of 10-ish people there were one or two who had even a vague idea what a kernel is (they were engineering students, actually), and were suitably impressed, but the rest of the group simply offered...

On constants in algorithmic complexity, and comparing apples to oranges

Can an O(n!) algorithm be faster than an O(1) algorithm? Yes, of course it can. So I came across an interestring and almost trivial programming puzzle: given an arbitrary word and a dictionary of words, find all anagrams of the word present in the dictionary. At first I didn't care to think about the best solution for the problem and I just wrote a trivial, brute force solution as a proof of concept: from itertools...

How about a Digital Price Tag?

How about adding intelligent information to our price tags? BitCoin and DogeCoin (and others) already have payment URLs, but we can do better than that! The easiest way to offer something for purchase using cryptocurrency is simply to link and/or describe what you want to the buyers on your web shop. You don't really need middle-men and merchants to do so: it's easy to create your own "BUY" link yourself. Option #1: payment URL's Most people know...

Can UBI be tested with Digital Currency?

This article on the discrepancy between consumers' lifestyle and deteriorating workers' rights has sparked a prodigiously long discussion on the Futurology subreddit (where the usual median length of the discussion is somewhere around 10 comments). Such topics now regularly venture into discussing the merits of Universal basic income (UBI), which as an idea was practically non-existent in the mindset of two or so years ago. An offhand

The Pirate party of Croatia - what went wrong

The context of this article is that it's a sort of a post-mortem written from my own perspective of what went wrong in the Pirate Party of Croatia, now that I've left it. I've been one of the founding members and after more than two years' work I must admit that the number of accumulated problems has surpassed the level at which we can be productive, and that I cannot aid in solving them. Simply put, people in general (at...

The Needle Search Server - pre-alpha

I've talked about my new project, the Needle Search Server before - it is supposed to be a light-weight full text search server written in C++ and using LevelDB for storage. I've arrived at a point where the code actually does something useful and I want to talk about it some more. Of course, you will need to fetch and compile the code yourself and once you get over that hurdle, you...

A brief history of computing in error messages

Technologies and systems come and go but error messages stay. Of course, this is because errors are (or were, before exceptions) signified by integer error codes and libraries maintain, among other things, dictionaries of human-readable messages to go with such codes. Most system programmers, or any other programmers which still dwell on C instead of using something less close to the metal, can remember a subset of those codes simply because they've seen them often enough. I first...

Dear Google, make me a Phone

Dear Google, please make me a Phone which lasts 24h. Heck, even 18h would be a marked improvement since no phone that I've used lasts me even 12h. You see, I actually use my devices, they are not sitting in my pocket among the lint. Also, I don't use them for playing games or mining Bitcoin, but for reading and communicating. Please, dear phone industry, get your collective heads out of your asses and make a phone which: Has a CPU...

Starting a new project - Needle

I have recently built an "Open Government" service which takes all the documents from the official Croatian government gazette which, among other things, publishes laws, changes to laws, decisions of the Constitutional court, etc. and indexes them, offering two new services: better full-text searchability and data "push" approach, allowing users to "subscribe" to arbitrary search queries and get notified when there are new documents published which are matched by those queries. Though these documents are pro-forma published on-line at...

Switching my blog to Pelican

After using my own blog platform since 2008, I figured it's time to give up on maintaining yet another piece of software just form my own use and switched this blog to Pelican. It's the popular static html site generator (the old blog system was also static) and I hope it will serve me well for years to come. I've imported old articles and comments, and new comments can be added by using Disqus. This makes my blog...

pkgng goes official in FreeBSD

As seen on the announcements list, pkgng is now officially up and running for FreeBSD! The infrastructure had some rough times before and the deployment was prolonged a bit, but now it's finally there and the old pkg_* tools can be put to rest. The preferred format for referencing repos has changed a bit, so if you have old config files, be sure to update them! Instead of specifying PACKAGESITE in the main pkg.conf, now each...

Why UFS in FreeBSD is great

ZFS is of course the rock-star file system in FreeBSD, with numerous features and new ones frequently coming in, but UFS is also a pretty solid deal which is perfectly usable for a wide number of tasks. Here are some of my favourite UFS features. #1: UFS is a pretty old file system which has been continually incrementally upgraded though the years. One of its best features is that its memory usage...

Apache 2.2 and Perfect Forward Secrecy (PFS)

Update: apparently (I haven't tested it yet), Apache 2.2.26 finally supports ECDH cipher suites! The remainder of this blog post is not as usable any more and you can simply use some common SSLCipherSuite lines. Using modern Perfect Forward Secrecy (PFS) cipher suites with #Apache 2.2 and #OpenSSL is not really possible in the general case. The best you can do is enable some DHE suites instead of...

What I like about the Nimrod programming language

Nimrod is a relatively new programming language which I find very interesting mainly because it is a compiled language and has a syntax which is a combination of Python's and Pascal's syntax, the two languages I find most aesthetically pleasing, with a small sprinkling of other influences, like JavaScript. The language and its community are growing, there is an increasing number of available modules (like Jester, for web apps), and it's currently on the...

pgrep - another little known FreeBSD command

Have you ever wanted to perform an action on a large set of processes which are forked from one another (or started from the same binary in some other way) but were stopped by the need to use awk or something else to parse the PIDs from the output of ps or a similar utility? Pgrep allows you to do just that! The pgrep utility is actually not a program developed specifically for FreeBSD...

GEOM_SHSEC: A shared secret disk drive GEOM module

GEOM_SHSEC is one of the less frequently used GEOM modules from FreeBSD, but it is actually pretty interesting. It combines several drives (or any other entities which are presentable as GEOM devices - including USB memory keys and files) into a single virtual drive which has the property that all its constituent devices must be present and available before its contents can be accessed. For example, you might have two USB keys which both need...

Viber voip+messaging app working on FreeBSD under Wine

A friend told me that Viber (a free VoIP + messaging application for smartphones, similar to Skype) now has a Windows application and it works for him on Wine on Linux, so I just had to try it out on FreeBSD - and it works. Surprisingly easy. I guess even Wine is mature enough on Windows :) Viber has been a bit of a troublesome app for me on Android, with frequent lockups and...

People will be people

I am frequently amused by all the recent happenings around Snowden and the revelations of privacy breaches by various governments. Before going on, I'd like to say that I don't offer any answers here in this text, just commentary, so it is probable you won't learn anything new from me. I'm writing this because I have an itch to scratch... Again, I feel the need to point out that I'm aware...

OpenVPN on FreeBSD

OpenVPN is probably the most popular semi-non-standard cross-platform VPN solution, with a large number of users and a pure userland implementation. It's pretty easy to set up, but I often forget certain steps so here's a tutorial for me to rememeber in the future :) I call OpenVPN semi-non-standard because it uses its own protocol instead of L2TP, IPSec or something other blessed by a RFC. OpenVPN takes care to be secure and offers some...

BSDCan and FreeBSD DevSummit 2013

It's that time of the year again - time for the biggest, bestest gathering of BSD geeks from around the world - BSDCan 2013. It was great to see old friends and new faces, exchange ideas and talk about the bright future. The main event was traditionally precceded by two days of tutorial and the "Developers' summit", a sort of a general assembly of FreeBSD developers, used to exchange ideas and often also directly work...

FreeNAS vs NAS4free

I've (finally) tried both FreeNAS and NAS4free and I'd like to share some thoughts and experiences. Both of these are "NAS-in-a-box" products intended to be installed on computers with a large number of drives, which they will export to the world in a variety of protocols. Both are based on FreeBSD, both fully support ZFS and they even share a common history. The tl;dr of this post would be something like "yes, they...

FreeBSD 10 news

There's been a lot of maturing of technology for FreeBSD 10 - lots of new features which make this release the most exciting one in years. Here are some of my personal highlights. There's been a lot of development in the field of virtualization. FreeBSD has almost neglected "full" virtualization technology (relying on home-brewed jails/vnet), which made it a poor choice for modern deployments, but that is about to change. New stuff which will (probably, but...

Parallella on Kickstarter

Kickstarter is a great thing - it allows projects which limited appeal to be succesfully financed, which helps them succeed. One such project is the Parallela. It is basically an ARM-based highly-NUMA computer with 16-64 cores which can be used both to teach parallel programming and actually do some useful work with very little electrical power. The numbers cited as on the on the order of 45 GFLOPS/watt for the maximum configuration. As the...

Using pkgng in real life

I have been using pkgng on a few machines now and I'd like to share some thoughts about how it behaves in real-world situations. Overall, I'm very happy with it and it's immensely better than what we had before. There are some rough edges which need to be solved but those are mostly a property of the ports system itself rather than pkgng. I'm practically ecstatic about pkgng for one huge reason: no more waiting...

Neil Armstrong was on the Moon!

Neil Armstrong walked on the Moon (and that's Moon with the capital M, it's its name) on July 21st 1969. He was 38 years old. That is way before my time. I'm 30 now and I didn't, couldn't witness it myself. My father, though, is of the generation which did witness it, in a live stream - what they used to call "world-wide broadcasts" before Internet made it obsolete - and he told me about it. Nobody...

pkgng - best thing since sliced bread!

FreeBSD (and BSDs in general) traditionally have source-based upgrades and installs which extends to the third party software collections - ports or pkgsrc and similar. This is all fine and offers unprecedended flexibility when tailoring system to specific needs, but sometimes this flexibility is less important than ease of use or time savings which can only be achieved with binary packages. Enter pkgng, the next-generation binary package management system by Baptiste Daroussin and...

Secure logins without SSL?

SSL is good, SSL is great, SSL does a lot of things for a lot of people, but sometimes you just need a secure login method without the overhead of using SSL. As a first step, I've created a simple library, rsalogin, which uses a JavaScript RSA implementation and allows the client and the server to exchange data in a relatively secure manner.  It borrows the approach of verifying server keys from SSH: the...

Writing a GEOM GATE module, part 4

To wrap up this small tutorial on writing ggate modules, I'll describe in brief how the simple implementation of ggvd works. Unfortunately, it has a major limitation in functionality due to a FreeBSD bug but it still may be good enough as an example. Here's a list of tutorial parts: First part - Describes what GEOM and GEOM GATE are Second part - Describes some more on how GEOM and GEOM GATE work, discusses sector...

Writing a GEOM GATE module, part 3

This is the third part of my supposedly short GEOM GATE tutorial. In this part I would like to show how to use ggatel and what do I want to do as a brand new GEOM GATE module. If you missed the previous parts, you should read the first and the second part before this one. Here's a list of tutorial parts: First part - Describes what GEOM and GEOM GATE are Second part -...

Writing a GEOM GATE module, part 2

This is the second part of my short tutorial on writing a GEOM gate module for FreeBSD. If you missed it, see the first part for an introduction. In this part, I'd like to talk about some of the mechanisms, ideas and constrains in GEOM and GEOM GATE (ggate). GEOM deals with the type of devices which were formerly known as "block devices". Though the distincion (between "block" and "character" devices) is no longer there, for...

Writing a GEOM GATE module, part 1

For various reasons, including the common good, I'd like to write a short-ish tutorial about writing GEOM GATE modules for FreeBSD. I hope to do this in a series of a few blog posts, dealing with the steps of writing such a module, this being the first post in the series. Finally, I'd like to do it while also creating a usable module, so this will not be an academic excercise but (hopefully) useful work. I'd...

ThinkPad non-OEM batteries?

I have a nice little ThinkPad Edge 13 and it's a great machine, really - amazing cost vs value. I've happily been running it for more than a year (so yes, the basic warranty has expired) and I've attempted to get a new battery for it - which failed in the usual insidious way: apparently the machine won't accept non-Lenovo batteries. This is a big #fail for Lenovo and means that this will be...

BSDCan 2012 - Day 2

The second, and unfortunately the last day of BSDCan was filled with interesting talks, again with much overlap. There are simply so many interesting things going on in FreeBSD that all of them simply don't fit in just two days of conferencing! From all of those, I'd recommend (even though I wasn't able to attend some of them) the talks on netmap, ZFS, AWS, pkgng and IPv6 security -...

DevSummit 2012 day 2 and BSDCan 2012 day 1

The second day of the DevSummit continued with interesting technical discussions in the Virtualization track, which was paralleled with the Teaching OS Courses track and the Administration and Toolchain tracks. The BSDCan day began with an epic bagpipe performance followed by full four tracks of highly interesting topics - unfortunate as there is much overlap. I gave my talk on Bullet Cache which describes some of the more interesting technical aspects...

BSDCan 2012 - DevSummit

Another year - another BSDCan! It's very nice and even comforting to see such a large number of familiar faces again, and even more as the ranks are filled in by fresh new developers. The conference and the Developers' Summit before it promise a great program and a great time for the BSDers. FreeBSD as a project is blessed by a comparatively low number of kooks and freaks - which is probably a natural consequence of...

Pirates ahoy!

I'm blogging a little bit less frequently then previously simply because I have less time to do it - but I'm happy to say it's for a good cause which also makes me personally happy: I've been involved in bootstrapping the Croatian Pirate Party.   It's been a wonderful few months, seeing something like the Pirate party grow literally from nothing. I've met...

Google Summer of Code

Time flies! The last year's barely passed and here we are again - another Google Summer of Code. Even though the deadline for student's submissions is less than 24 hours ahead, I'd still take the opportunity to call anyone interested, anywhere in the world, to submit a proposal - it's fun to participate! Last year I've (co)mentored the BFS port and the latest attempt to bring FUSE to be usable on FreeBSD. Of those, the BFS...

Kindle text editor bounty

I love my Kindle 3. It's a great little gadget which has increased the amount of books I read - which is by itself an outstanding accomplishment. I feel I don't need to describe its many benefits here because the interested reader either already has it or knows how to use Google well enough to find out all about it. But, there is one very significant feature from Kindle which I very much miss -...

Dealing with Trac performance problems

Trac is a great little web tool for both developers and random collaborators - it's a wiki integrated with a Subversion browser with simple project management tools and extensible with a lot of different plugins. Unfortunately it's also slow. I've had to deal with an Internet-facing Trac wiki and came up with some solutions which lessen this problem. While Trac was great when I introduced it as a simple project management aid in a team...

Bullet Cache use cases, Part 2: Data sharing

In addition to simple data caching, there are some interesting advanced features made possible by Bullet Cache's unique features. Data sharing between applications (or between application instances) is a very important one, especially for the PHP environment (and other CGI-like environments). This post is a part of series on Bullet Cache use cases. Previously in the series of posts about Bullet Cache: Introduction Installation and testing The C API tutorial RC1 announcement Lessons leared...

Bullet Cache use cases, Part 1: Data caching with record tags

Use cases for Bullet Cache are numerous - on the one hand it is a very convenient (and fast) memory cache server, but on the other it implements some advanced features which make it applicable in surprisingly many different scenarios. This post starts a series of descriptions on some of the real-world use cases which fit Bullet Cache suprisingly nicely, and some of these are already described in the Bullet Cache User's...

Donate to the FreeBSD foundation!

I'm on a trip so I'll be brief: Donate to the FreeBSD Foundation! It supports and funds vital FreeBSD development! See more at http://www.freebsdfoundation.org/donate/ . http://www.freebsdfoundation.org/donate/ #1 Re: Donate to the FreeBSD foundation! Added on 2012-01-04T08:46 by dot Hi,I'd like to donate but it's too complicated at least for me.They want $, i have €-s. They want/like credit card, i have debit. Rumors are, that i could donate with...

Lessons learned with Bullet Cache

I'm preparing for the 1.0 release of Bullet Cache and have squashed the last (known) bug which plagued it, so I'm cautiously optimistic that it deserves the "1.0" label. It's been very fun working on it and though none of this is terribly exiting news, I'd like to share a few things I've encountered while making it... #1: Fast hardware isn't When it comes to performance, there is absolutely no point in emphasizing one hardware...

Encrypted PostgreSQL data types

The last few nights I have been working on something very interesting: encrypted data types for PostgreSQL. The goal here is to introduce transparent data encryption for applications which need to protect "data at rest", i.e. while the data is stored in the database within the file system. I didn't find any such mechanism (pgcrypto only offers security primitives which can be used on the application side) so I wrote pgenctypes. The issue...

Call for testing the BSD Hypervisor (BHyVe)

Those who were present at BSDCan 2011 have probably seen or heard about the unveling of BHyVe, the native hypervisor for FreeBSD. FreeBSD is very much lacking virtualization features (not counting jails) and this is in any case excellent news for the project! Interested users are now invited to test it! BHyVe currently has some good sides and some bad sides. The good sides are that it finally (!!!) exists and that apparently it...

Bullet Cache - RC1 and main features

I've just uploaded the first Release Candidate version of Bullet Cache! It is basically feature complete and done, and I'm happy to say that it looks like I have a small number of users and also some feedback on the project - so keep it up :) At this point I'd like to shortly talk about what made me write Bullet Cache - which also leads to why it was done the way it...

My digital life

I went to the customs office to pick up a package, ended up told that I need more papers than I brought in, specifically a PayPal bill printout which I didn't have. Now, the usual thing to do would be to return home, print it out, then return back to the customs office and proceed from there. What I did instead was amazing - from the point of view that almost nothing of that infrastructure...

FreeBSD's /rescue directory and system recovery

One of the many neat things FreeBSD uses to increase the operating system's resilience is the /rescue directory which is present by default and contains basic utilities and commands which can be used in case one or more of the default binaries gets corrupted or lost. In fact, binaries in /rescue are also statically linked so they can be used even if a large part of the system fails. Basically, if you have init and...

Bullet Cache - The C API tutorial

This post in the Bullet Cache series introduces the primary API to the cache server, implemented in the for of a C library. This is important as it allows for maximum portability across different application environments while at the same time retaining maximum efficiency and performance. All of the standard benchmarks were done using this exact library without any tweaking. The standard distribution also includes the PHP API which is an almost...

Bullet Cache - Installation and testing

I've written previously about my Bullet Cache server, and now I'd like to continue with basic installation, startup and testing. The server itself is written in a combination of C and C++, and should be usable on most modern Unix-like systems. Bullet Cache is made to run primarily on FreeBSD but I try to keep up Linux compatibility (which is actually pretty easy as both are reasonably POSIX friendly). As the first step, fetch the

Bullet Cache - Introduction

I am going to take the opportunity in the next few blog post to advertise my open source project - Bullet Cache, which is almost finished. Think of Bullet Cache as [*] "Memcached, only better" :) It offers some unique data query options and is blazingly fast! And good news for everyone who's asked: as promised, I've licensed it under the BSD license! I'm announcing the availability of the 1.0-beta1 version of Bullet...

"Nobody got fired for choosing IBM" - what is COTS?

I've just been to a meeting about what should have been a mundane and easy topic: there is a need for the update of a technologically old Web system of a government agency and through word-of-mouth it was thought that my group could do something good about it. As it turned out, that's a minor part of the work (and doable) and the rest of the talk was eerily Dilbert-like, practically the real-world incarnation of...

JSP Performance - I have stumbled upon an oxymoron

As one of the commenters figured out, I made a fatal error in not running Tomcat as a standalone server, which had a horrible impact on its performance. After fixing this, the performance results between PHP and JSP are very near, probably within measurement error. So I'm retracting this too-hastily written post and apologize for any grief it caused to Java users :) It is the year 2011 and the "Java is slow" notoriety still...

Some simple Xen performance measurements

When it doesn't do IO, virtualization is pretty good performance-wise these days. Here's how Xen looks like  on FreeBSD 8-stable AMD64 XENHVM (pysysbench). Without Xen (no vrtualization): Running 8 benchmarks on 2 threads for 10 seconds eachRunning benchmark: Warmup-Ignore ... score: 230.2 average: 115.1Running benchmark: Hash-SHA256 ... score: 231.2 average: 115.6Running benchmark: Hash-SHA512 ... score: 364.4 average: 182.2Running benchmark:...

Isaac3 - yet another release

It looks like I'm releasing a new version of my ultra-compact single-file image gallery approximately once every two years :) It is extremely simple and definitely lacks modern features (it was started in 2005!) but if you use it, enjoy! Apparently I've been commiting cool new things to the repository without making a release, so this one packs a number of new features: Better (not nearly perfect) compatibility with PHP 5.3 Better concurrency via timely session lock...

Testing the (new) FBFS scheduler

As some of the regular readers may know, I'm mentoring a Google Summer of Code project that aims to bring an experimental light-weight scheduler to FreeBSD. There are many differences between Linux which is the original scheduler's initial implementation platform and FreeBSD, so the port will be more like an reimplementation of some of the ideas. This project is important because it brings a new perspective to the problem of scheduling...

Google+ and scalability

In a curious twist, it looks like Google+ currently causes more trouble for Google than it's worth it. I have no doubt that the problems will be solved soon (if nothing else succeeds - they can always throw more hardware at it), but it's instructive to say the least. As I'm writing this, all non-search services of Google I've tried are broken to a greater or smaller extent, starting with Google+. I suspect this is because Google+...

My little niece

My "little" sister gave birth to a brand new baby girl about two months ago, and is now going through the usual baby-raising process, invloving a lot of crying, pooping and not much sleeping. The baby is - cute, like all babies are. I'm much drawn to the process where the baby is "Awakening" (yes, with a capital A - you know what I mean). It still has a long way to go but it...

Bullet cache

I have written before on mdcached - my not-so-pet-project I've been doing over the years, which I'm looking forward to bring to a really usable state in the coming weeks. I've cleaned it up a lot since March and I'm confident that the core of the system is correct and reasonably bug-free. I am renaming it to "Bullet Cache" but I am still calling it "alpha" code as it is not yet...

World IPv6 Day - How to run 6to4 on FreeBSD

As you know, or should know, today (or tomorrow, depending on the timezone) is World IPv6 Day. What this means in practice is that a lot of top-tier IT businesses will make their web sites available via both IPv6 and IPv4, at least for a day. If you want to join in testing but your ISP is following the rest of the herd and isn't doing anything with IPv6, you can use various methods...

Progress in CPU architectures

I got access to a Sandy Bridge Xeon server for a while and decided to try some benchmarks on it. Usually, I'd run unixbench from the FreeBSD ports, but - it's old. It is very, very old and the only reason I still use it is convenience. So, having some time, I started my own multiprocessor-friendly benchmark suite. Yup, I could have named it "yet another benchmark suite" since there are so many of them....

Two FreeBSD Summer of Code projects

FreeBSD has 17 Google Summer of Code projects this year and most of them look very interesting! As I am involved as a mentor or co-mentor in two of those, I would like to describe them here. First of these is the port (or reimplementation) of the BFS scheduler from Linux, and it's blogged about in detail here. It is a very experimental project which aims to find out if having a scheduler...

BSDCan 2011 and DevSummit

Once a year, Ottawa (in Canada) becomes the hottest place to be to discuss BSD-related projects, at the event unsurprisingly named BSDCan. Hidden within this event is the FreeBSD developers' summit which is itself a great opportunity to see all the people I've communicated over the year, talk and exchange ideas. This year's BSDCan / DevSummit was one of the better ones, measured in terms of ideas and projects presented; it was definitely a success! For...

LibreOffice 3.4? Why?

Apparently, the LibreOffice is project set to release version 3.4 ... with these changes. I find it very WTF-worthy that these changes should amount to a whole new version instead of just 3.3.3 or something like that. Ok, I'll play a fortune teller and say that, seeing the regressions in LibreOffice 3.3 compared to Oracle's OpenOffice (scrolling bugs, crashes), and how much of the features In LO 3.3 were actually developed by Oracle and just...

My brief experience with SSDs on the desktop

This is the first time I've tried to used an SSD on a desktop, and it was primarily to try to speed up Windows booting and application start. In short: don't bother. I experimentally bought a small but relatively high-end SSD: Mushkin 60 GB Deluxe, with the Sandforce SF-1200 controller, to be used as the OS disk drive. I could have bought a bigger one but really didn't wan't to experiment too much on my...

Firefox 4 in FreeBSD ports!

Without any delay at all, Firefox 4 is available in FreeBSD ports! Big thanks to everyone involved! So far I tried Acid3 and WebM movies and they all work fine... Font rendering looks much improved also! #1 Re: Firefox 4 in FreeBSD ports! Added on 2011-03-23T19:43 by Ronald Klop Loving it! I can watch more and more movies on the net without flash....

Google Summer of Code 2011

As probably everyone knows by now, Google Summer of Code 2011 is announced! As in previous years, FreeBSD is expected to have a presence and this is a great opportunity for every student interested in FreeBSD to get involved and get payed doing so! FreeBSD has a pile of ideas page and I'd like to promote some of those I think are interesting. I have been in GSOC a number of times but I...

mdcached

For some years (many more than really necessary) I have been workin on and off on a pet project of mine, mdcached - a cache server similar to memcached. I've figured it's time to finish it and bring it to usable state now, so (I hope) I'll be writing up the progress here. For now, I have some really nice benchmark results which show off its strengths compared to memcached. mdcached (the name is...

New micro-servers

As previously noted, I have an interest in running large-scale services on equipment which would be called underpowered by many, so I find that this 256-CPU-cluster-in-a-rack is in many ways a dream machine. Of course, I don't think of them as underpowered but as efficient for the task, scalable and finally, green. I would go as far as to say that if there need be clouds, these are the type of...

Bad experience with DealExtreme

Like probably everyone who is reading this, I occasionally buy gadgets online, usually via eBay. I've heard of DealExtreme before and some people I know had ok experiences with it so I decided to try it - that was last year in December. Due to some bad luck and apparently horrible organization of DX, I still haven't received the things I bought. At this time, effectively I've been cheated for around $400 by DealExtreme. At least...

Direct file versioning

Another thing I've wanted for a long time is integrated file versioning (if possible coupled with pervasive auto-save). It looks like the new Mac OS X versions will have it. Apple apparently has the knack for getting things right. I have no idea how the feature is implemented in MacOS X Lion but I would do it with multiple file streams, with a simple addition of an API which makes one of...

Computers in your hand

Something I've wanted for a long time is finally happening, though slowly and with shy experimental feel to it. "Smartphones" nowadays are essentially computers with more computing power than "real" desktop workstations 5 years ago: remember that it was only in late 2005 that dual-core Intel CPUs were coming fresh out from foundries and with initial clocks lower than 2 GHz. Now, all we need is good (and if possible wireless) peripheral connectivity...

A couple of Anti-Cloud links

Here is a couple of examples where "clouds" are a bad idea (basically a "RISKS" take on clouds): Google blames bug for Gmail deletions RIP, Danger, 2002 - 2011: Microsoft axing service on May 31st, T-Mobile promises an 'easy transition' It's not that the basic idea of the "cloud" is bad in itself but the problem is that it introduces unneeded fragility in frequently used Internet services for the end-users. At best, it...

ZFS v28 in FreeBSD 9-CURRENT!

As has been expected, previously announced and tested, ZFS v28 has been committed to FreeBSD HEAD! New features include: RAID-Z3 (triple parity - one more parity drive than RAID-6) Deduplication Better recovery support during import (forced log rewind, read-only import) Snapshot-level diff (like regular diff but working on file systems) zpool split (split a RAID-1 / mirrored set of drives into separate / independant zpools) As always, testers are welcome! ETA for backporting to 8-STABLE is set at one month, but...

OpenOffice, LibreOffice: why do they suck so much?

LibreOffice is a recent fork of OpenOffice, made to counter Oracle's mistreatment of the neat stuff they bought with Sun. Whatever Sun did or didn't do, its policy of openess is something so good and unique for a large company that I doubt we will see it again any time soon. Anyway, LibreOffice has big plans. They are trying to set up a foundation to ensure resources and governance for future software development...

Who uses FreeBSD?

One reason why I maintain this blog is to do some advocacy for FreeBSD - attempt to spread the word about new developments and generally promote it. It's not an easy job as the system constantly struggles for popularity aside the much more popular Linux (and on the other side - older Unixes and Windows). I've accidentally come across this page on www.freebsd.org listing some popular FreeBSD users but I think it badly needs updating. Are...

FreeBSD Status Report - Q4 2010

It's time for another FreeBSD status report! Among the many news I'd like to highlight: Another effort towards a new installer - BSDInstall. Good luck :) Non-executable stack support ZFS v28 TRIM Support for UFS Xen domU work enabling use of FreeBSD on EC2 Read the whole report for details! FreeBSD Quarterly Status Report Introduction This report covers FreeBSD-related projects between October and December 2010. It is the last of the four reports planned for 2010....

FreeBSD on 4K sector drives

All major FreeBSD filesystems support 4K sectors (UFS, ZFS, ext2), and so does the lower level - GEOM - but currently there's an issue of communicating this configuration between all the layers. A part of the problem is that the current drives (and the situation will probably not change during this new decade) advertise two sector sizes: both 512 byte and 4K, and the system needs to correctly interpret them. All this will be resolved...

FreeBSD Foundation donations

When money abounds, you buy office furniture and gamble with investments, when it's tight, you do something with it. The FreeBSD Foundation has done a lot this past year: Provided $100,000 in grants for projects that improve FreeBSD in the areas of: - DTrace support - High availability storage - Enhanced SNMP reporting - Virtualization and resource partitioning - Embedded device support - Networking stack improvements Allocated $50,000 for equipment...

Resource reservations and local DOS

An interesting post on the stable@ mailing list by Matt Dillon illustrates why are jail resource limits and other such measures important on todays machines. Actually, this also illustrates one of the "pro" arguments for virtualization. The post is very informative and I'm quoting it here: One of the problems with resource management in general is that it has traditionally been per-process, and due to the...

The wonderful en_DK locale

English is generally great for communication, but there's a certain awkwardness when it comes to date/time formatting rules. Specifically, I live in a country which, like most Europen countries has a "day, month, year" native ordering and %d.%m.%Y formatting, but aspires to one day adopt ISO 8601. The default en_* locales don't support such a patchwork of formatting, so there are hacks to get around that; en_DK is one of those hacks, used...

How to play Osmos on FreeBSD

The "Humble Bundle" project brings indie games to the masses by allowing people to pay any amount to buy a whole bundle of games together. Since they are developed by small developers, many of them (probably all) also work on non-Windows OS-es. I've bought the bundle and wished to play Osmos on FreeBSD. My configuration is AMD64 with an ATI card, usually a terrible one for gaming on non-Windows. Osmos is one of those...

ZFS v28 imminent

Another ZFS patch was announced minutes ago, bringing ZFSv28 and all its goodies to FreeBSD. Testers are wanted! Update: A patch for 8-stable is also available! Some of the new features available with the patch are: - Boot support for ZFS v28 (only RAIDZ3 is not yet supported). - Various fixes for the existing ZFS boot code. - Support for sendfile(2) (by avg@). - Userland<->kernel compatibility with v13-v15 (by mm@). - ACL fixes (by trasz@). Unfortunately,...

ZFS in Debian

Somehow I hear the "ha-ha!" soundbite of Nelson from The Simpsons when I think of this topic: ZFS is coming to Debian! Of course, it will be arriving there via the FreeBSD kernel There are apparently some "license wars" between GPL and CDDL (the GPL doesn't allow inclusion of CDDL code) so this is slowing down or precluding the inclusion of ZFS in GPL-only systems....

Another reason against "clouds"

Why not set your data into a big company's cloud? Also because of this: Amazon kicks out WikiLeaks under government pressure. Since any dubious "damage" has already been done, this is simply a bad gesture. Did I mention I'd like distributed p2p HTTP? :) #1 Re: Another reason against "clouds" Added on 2010-12-02T14:34 by Anonymous Coward Yeah, net fragmentation is such a great idea! Maybe those freedom...

The fragmented future of the Internet - distributed p2p DNS?

It looks like the recent events with WikiLeaks and the fight over DMCA have provoked an incredible amount of new ideas in secure distributed computing. Basically it all comes down to the question of centralized maintainance and governance of the Internet or a decentralized and very possibly anarhistic one. Lately, I'm much more inclined to think the anarchistic future is not only better but probably also unavoidable. In the recent months, these things have happened on...

Donate to the FreeBSD Foundation!

As it does every year, the FreeBSD Foundation is gathering donations to help support the FreeBSD project. As FreeBSD doesn't have large sponsors, it relies on little donations and volunteer work to keep it going. A letter from Justin T. Gibbs of the FreeBSD Foundation: Dear FreeBSD Community, As the year is winding down I'm writing this note to remind you of the motivation behind the FreeBSD Foundation's work, its benefits to you, and to ask for...

WTF Mozilla?

It looks like I'm missing something here so I'd like someone to explain to me how Mozilla Foundation, a company into which Google dumps nearly $100 million a year, gets to produce Firefox (for which the only reason for using is that it sometimes sucks less than the competition), a mostly (still!) unifinished Thunderbird 3, and a couple of academic excercise-style projects? Seriously, a company in which Google dumps nearly $100 million a year produced...

UFS read-ahead

After 10 years of it being conservatively tuned, I've recently increased the default read-ahead (vfs.read_max) in FreeBSD from 128 KiB to a whopping 512 KiB. And of course, I have received an e-mail from a concerned developer asking if that is perhaps too high :) How much impact can read_max have I'll try to illustrate on this excellent example on a machine I'm currently configuring. The machine is a small but modern server with two 7200...

About that demon...

There was a regular one-sided flamewar these days on the freebsd-questions@ list about the BSD mascot - Beastie the demon. The question is not the least new and it is asked regularily with approximately the same answers each time. For my part, the answer is that the picture is here because of a chain of events: 1) "demon" is a colloquial name for system processes (mostly "servers" like web servers, etc. but other important processes...

The Anti-Cloud

I like big storage, I work with virtualization in its various forms so that all the hype about "cloud computing" sounds like "well, duh!" to me. Still, I think it's wrong to look at it as the possible future of IT in general. We have worked hard to move away from centralized computing and I think we should not go back. As I've written previously, I want just the opposite of the current "Cloud" fad...

FreeBSD Status Report - Q3 2010

A new FreeBSD quarterly status report has been published! Among the many reports, some selected new statuses are: Status of Summer of Code projects News on Capsicum security capabilities ZFS, GELI and HAST news Userland DTrace CLANG replacing GCC in the base system New TCP congestion control algorithms Support for USB 3.0 Read the whole report for details!   FreeBSD Quarterly Status Report Introduction This report covers FreeBSD-related projects between July and September 2010. It is the third of the four reports planned for...

Colored snake!

Probably not many users noticed, but some time ago I committed a hopefully useful (but always shiny!) addition to the "snake" console screensaver - now it contains the system load numbers and is color-coded per system load :) Just to make this clear: this is probably a completely useless piece of code but I just feel giddy about it :) The snake is colored like this: No load (first load number = 0) :...

vfs.hirunningspace and disk write latency performance

A while ago I increased the default value for the vfs.hirunningspace tunable - which greatly helps with performance when the disk system supports tagged queueing (e.g. NCQ), allowing many more requests to be offloaded into the controller and/or the drive(s). But deep queues bring their own problems, especially in pathological cases. Vfs.hirunningspace basically controlls how much write data (in bytes) can be outstanding for write IO requests. This is orthogonal to synchronous / asynchronous writings...

The obsolesence of WYSIWYG

When is the last time you used the term "WYSIWYG" seriously? I was just randomly thinking about the history of computing and it occured to me that it is practically an obsolete term today, mostly because everything *is* WYSIWYG. ... Except of course it's still relevant in some specialized areas like TeX, which are *not* WYSIWYG. And possibly wikis. We really have advanced. #1 Re: The obsolesence of WYSIWYG...

Sixty Symbols / Nottingham Science

Almost since they started making them, I've been watching these science videos from University of Nottingham in which their resident professors, staff and guests discuss and demonstrate various topics in physics, chemistry and astronomy - they are absolutely great! If anyone is trying to figure out how to make "distance learning" succeed in practice - this is it. When joining the university I basically had to choose between Physics and...

Adventures with FOG project, aka "All software sucks"

[this post is basically me venting after an all-nighter, it's way too harsh than it should have been] An alternative title to this could be "if something looks too good to be true, it usually is." We needed to uplift 80+ desktop PCs from a pile of cheap parts to usable state in about two weeks. Nothing to it, right? Mass installations and cloning are done every day, right? Well, no. I basically had two projects to...

So how is FreeBSD 9 shaping up?

It's still early to talk about FreeBSD 9.0 release but so far there have been some interesting developments in the systems and a nice core featureset is shaping up. I'm still maintaining the "What's cooking" page and this post is basically an (incomplete) summary of it at this point in time. Of course, in addition to these features, there are non-stop modifications to all parts of the system, from drivers for new hardware to overall...

OpenJDK vs Diablo JDK performance

"DIablo JDK" is the "official" FreeBSD-foundation-distributed version of Java, presumably tested with Sun's compliance tests. OpenJDK 6 is the new-ish open source Java base, which will if recent events are any indication, soon die under Oracle. OpenJDK is a much newer "minor" version of Java and as such offers some performance gains: betelgeuse:/usr/local/share/java/scimark2> java -showversion jnt.scimark2.commandlineopenjdk version "1.6.0"OpenJDK Runtime Environment (build 1.6.0-b20)OpenJDK 64-Bit Server VM (build 17.0-b16, mixed mode)SciMark 2.0a

Google Instant search is not magic

In the last few days I have read several blogs, even technical blogs which should know better, treating Google's Instant Search like it's something otherworldly. Without actually analyzing how it's done, I think it's simply one of these things whose time has come. In other worlds, here's how I'd do it: I've done extensive work with caching in web applications and basically, I think the solution is in the intersection of the following trivial facts: RAM...

GNATS

I'm just idly taking a look at GNATS... While the popular sentiment is "I'm too old for this shit", GNATS is one of those things that just want me say "I'm too young for this shit." GNATS is an elderly GNU project which aims to be a simple bug tracker. And going with that definition, it succeeds. It is simple (too simple) and it is a bug tracker. One of its (former) ground-breaking features is...

FreeBSD 6.x EOL

A Security Officer bulletin has been issued recently to announce the end-of-line for FreeBSD 6.x - which is an opportunity to discuss how FreeBSD release cycle generally works (for those not familiar with it). Firstly, while it may look like yesterday for developers and users tightly in tune with FreeBSD development, FreeBSD 6.0 was released 5 years ago. FreeBSD has this great feature called "backwards ABI compatibility" meaning that you can take a program, and...

HP "LeftHand"

I've seen a HP "LeftHand" / StorageWorkd P4000 SAN device recently and got quite good impressions off of it. One thing that occured to me is - why didn't anyone try this before? Certainly both Linux (to lesser extent) and FreeBSD (to a somewhat greater) contain the pieces for it, and have contained for some years now. In fact, several people did such setups privately or internally for their companies but there was apparently...

Two ZFS performance patches' calls for testers

Martin Matuska has issued two calls for testers with patches that improve various aspects of  ZFS performance. The patches are basically ports of OpenSolaris' code and should improve write performance and stat() syscall performance. The patches are available for testing on 9-CURRENT and 8-STABLE. Interesting information for the first patch: The source states, that with this change they archieved a speedupof: "50% Improved OLTP Performance, 70% Reduced Variability, 200%Improvement on MS Exchange" and for...

Have an WD EARS drive or other 4k sector drive? Try this.

The industry is moving toward 4k-sectored hard disk drives - a bit late though but better late then newer. However, in the interim period where compatibility for 4k sector drives is not widespread, drives such as WD's EARS series are using internal translation to present themselves as 512-byte sector drives, leading to unexpectedly horrible write performance. As a band-aid I'm proposing an almost trivial patch to glabel which allows it to have a...

Encrypted notepad

A small personal project I've occasionally been working on is "Encrypted notepad" - a simple (really simple!) Notepad-like text editor whose only distinguishing attribute is that it saves and loads strongly encrypted files. Now, I have an Android version too :) The original purpose of enotes was to share sensitive data in our office but there are a million uses for it - from storing secret barbecue recipes in it to using it as...

Cheated on e-bay - microSD card by "Team Group Inc."

I've bought small electronics on e-bay before and had reasonably good experience - no especially bad products, but also no surprisingly good ones - which is ok, it is as expected and I was happy with it - until today. A new microSD card arrived (for my phone) which was branded - and priced - as class 6 (SD "classes" are basically performance indicators in MB/s), but behaves worse than a class 2 card. The...

FreeBSD 8.1 and status report

FreeBSD 8.1 has been released! This is mostly an maintainance release with few major changes: zfsloader added zpool version of ZFS subsystem updated to version 14 NFSv4 ACL support in UFS and ZFS; support added to cp(1), find(1), getfacl(1), mv(1), and setfacl(1) utilities UltraSPARC IV/IV+, SPARC64 V support SMP support in PowerPC G5 Some of the other interesting new features are: the ability to use VLAN tags with TSO, HAST high availability storage (disk replication) and new CAM-integrated SATA subsystem (not the...

PostgreSQL on tmpfs

Running a database on a memory file system serves a dual (and unfortunately not easy to deintertwine) purpose: testing database scalability and testing operatin system scalability. On the other hand, I did it just to see what the results could look like given an almost infinitely fast storage device :) The results of running PostgreSQL 8.4 on a 8 core (2x quad Xeon 5405), 4 GB machine, with FreeBSD 8.0 AMD64 with pgbench scale of "70"...

Good first impressions on Werkzeug

I haven't changed my opinion on Python's performance but it still is a great platform for doing small projects quickly. I've chosen Werkzeug for my project and so far I like it. It has just the right mix of low-level and high-level funcionalities for what I'm doing. Here are a few notes about it: Luckily, Werkzeug concentrates on one thing only and doesn't try to be everything for everyone - and what it does...

What to use for log compression?

I've made a patch for newsyslog to use xz compression since I just assumed xz would be better because of its structure. After all, at least it doesn't process files in individual 900 kB chunks. But I think I've found a case where xz may be worse then bzip2. This is an ordinary Apache log file, compressed with several compressors: -rw-r--r-- 1 ivoras ivoras 52300562 May 15 01:02 log -rw-r--r-- 1 ivoras ivoras 1972651 May 15 01:02 log.bz2 -rw-r--r--...

Python WSGI performance, or - is Python dieing?

I'm looking into starting an interesting little project and of course, one of the most fun phases of it all is choosing your technology stack. I'm going to do something a bit unusual, just because I can, but for starts, I've chosen Python/WSGI combination and within this palette of tools I've come across Werkzeug. It is a light-weight framework which I think will be perfect for the type of code I write - with...

All welcome ZFS v15 in FreeBSD

As announced before, ZFS v15 was successfuly imported into FreeBSD! For a time there was an option of importing just v15 or proceeding directly to v16 but the community has decided to first import the older version for reasons of stability and compatibility with Solaris 10 Update 8. Some of the new features are: 1. Stability - more mature ZFS code, comparable to Solaris 10 update 8 with latest bugfixes2. Compatibility - Solaris 10 update...

FreeBSD ABI compatibility

Julian Elischer, one of the kernel developers, pointed out that the new FreeBSD -CURRENT system cannot run binaries created for FreeBSD 1.0. While probably completely irrelevant for FreeBSD's deployments todays, the fact that you can actually run binaries created in 1994 on recent releases (in a chroot jail!) is a point of pride in the FreeBSD community. It is actually awesome! Hopefully, recent incompatibilities will be resolved as they historically were :) On 7/2/10 11:12 AM,...

Help test ZFS v15

The most recently ported ZFS version in HEAD and 8-stable (which includes 8.1) is 14, and now a call for testers was issued for v15. This new version brings in user and group quotas and there is a hint toward importing the latest ZFS version. A good thing about this patch / CFT is that ready-made minimal ISO images are provided for potential testers, which can be used to install new systems on real or...

Notes on pkg_patch

My Google Summer of Code project for 2010 is "pkg_patch" - a utility and a framework for working with binary patches to FreeBSD binary packages. It is necessary because the default way of dealing with 3rd party applications / ports in most BSD systems is to build them from source via various frameworks like ports and pkgsrc, and so far the binary package side has been mostly neglected, at least in FreeBSD. I strongly feel that...

FreeBSD's birthday

From the calendar: Jun 19  Charlie Root born in Portland, Oregon, United States, 1993 Jun 19  FreeBSD project born, 1993 Happy Birthday Charlie! #1 Re: FreeBSD birthday Added on 2013-06-19T06:57 by J Smith Happy Birthday Charlie & FreeBSD! Best wishes, #44BSD on freenode.org...

PHP, APC and shared memory

PHP is not exactly well known for its code quality (and I mean the interpreter, C code, not the canonically bad user-written PHP code). I think working with FreeBSD code spoiled me to expect more :) One "interesting" side of it is that even now, years after FastCGI has become the way to deploy PHP in production for any non-trivial sites, the most popular opcode / shared memory cache, the APC is broken in the...

LLVM in FreeBSD

The (approximate) date is set and LLVM will be imported into FreeBSD as an alternative base compiler, with the general intention of it being the official base compiler! There was much somewhat heated discussion about LLVM in FreeBSD but there are so many good things that will come from it that it would simply be unfortunate to miss them. In addition to these there are some more interesting stuff that can...

Wine on amd64

Users running AMD64 interested in Wine might want to take a look at the wiki entry and the download page. Of course, this is all completely unofficial. No warranties whatsoever. Your head might explode. #1 Re: Wine on amd64 Added on 2010-05-28T20:23 by Fernape I've been using it for a while, mainly to run Spotify. It works great. #2...

HTC Desire / Android impressions

Until I get to organize it into a proper post, this page will be a dumping ground for my impressions with Google Android and/or HTC Desire. Yay! I have a HTC Desire phone! Problems so far: Kernel and the base system support it but there is no high level support for USB or Bluetooth keyboards - none at all except for some commercial hackish addons! Kernel and the base system support it, but there is

On Facebook and (De)centralization

A somewhat surprising, but actually forseeable (and forseed) aspect of the WWW as an application platform is that it actually, in a way, ended up promoting centralization. There is only one Facebook, only one Twitter and Google etc, and they pretty much jelously guard their data. Sure, everyone can run a web server (there are a lot of them to choose from), but people use services and these come under individual addresses, not distributed....

Back to the HEAD

Just when I thought I can finally finish a whole project on a RELENG branch without touching HEAD (and thus do it directly on my desktop), there it is - with a "D'Oh" level of surprise I find that the pkg_install infrastructure is, yes, different enough in HEAD and STABLE. Specifically, pkglib has been separated out and with it came the "pkg.h" header. Which reminds me to be grateful about just how much finally having a...

Using multiple CPUs with gzip

gzip is ofcourse an ubiqutous compression program used in unix-like systems everywhere. Unfortunately, like most old utilities, it is completely sequential and single-threaded, which is a shame in a world where it is now becoming hard to find a desktop CPU without at least 4 CPU cores on it. There is a quick and dirty hack around it: gzip -2 < bigfile.vmdk | gzip -9 > bigfile.vmdk.2gz The theory behind the trick is to use two compressors,...

UFS journalling committed

Jeff Roberson has committed soft-updates journalling to 9-CURRENT! It enables a small file system journal which works in combination with soft-updates to eliminate boot fsck's. It is different from most other implementations of file system journalling in that it doesn't journal raw blocks but sort of meta-data about meta-data (hyper-data?). You can find more about SUJ in an earlier post. #1 Re: UFS journalling committed

FreeBSD Status report Q1/2010

The new FreeBSD status report is available! Find out about such goodies as the new CLANG compiler, network stack enhancements, CAM-based ATA, tickless kernel, UFS journaling (SUJ), embedded (ARM, MIPS, PowerPC) development and much more! The official announcement is at freebsd.org, but I'm mirroring it here for posterity: FreeBSD Quarterly Status Report Introduction   This report covers FreeBSD related projects between January and March   2010. Being the...

FreeBSD/CLANG ready for testing!

CLANG is a new C compiler built to work with LLVM. Getting away from gcc on the BSD side is kind of important (including, but not only because of GPLv3) so getting FreeBSD to compile and work with CLANG is a high profile project. FreeBSD on CLANG is ready for testing right now! Roman Divacky writes: ClangBSD is a branch of FreeBSD that aims at integrating clang (clang.llvm.org)into FreeBSD, replacing GCC as...

Support Chromium (Google Chrome) on FreeBSD!

Google Chrome / Chromium is the currently probably the best web browser out ther, especially in terms of compliance with new technologies' standards like HTML 5. It is also fast and secure, enabling better and safer browsing. The Chromium project is the open-source core of the Chrome Browser, and it is also available on FreeBSD! However, some more work is needed on it to stabilize and include it in the ports tree. If...

GEOM disk IO scheduler framework

A GEOM IO scheduler framework has been committed! The framework allows for multiple IO schedulers to be installed on top of GEOM providers (usually disk drives). As a consequence, potentially different schedulers can be installed on different drives. The work was done by Luigi Rizzo and Fabio Checconi. The traditional BSD IO scheduler is the "elevator" scheduler which basically attempts to sort the queue of IO requests in the order of increasing IO offset, attempting...

Of ports and men

There has been some discussion lately about if and how to "revamp" the ports system to make it more usable by general users, which was in part started by a post on PostgreSQL.org. Unfortunately there has been very little feedback from users themselves - which is probably a mistake, but also - there was very little feedback from the population (not a particularily small one) that is the cross-section of users and developers. Some...

Distributed SSH attacks

Oooh, how nice... another distributed ssh attack... need to get devious/er! Mar  6 05:45:25 lara sshd[67154]: error: PAM: authentication error for illegal user cz from ns.zircon.net.uaMar  6 05:46:50 lara sshd[67163]: error: PAM: authentication error for illegal user d from 193.178.147.249Mar  6 05:48:09 lara sshd[67170]: error: PAM: authentication error for illegal user daily from 205.234.239.234Mar  6 05:49:27 lara sshd[67181]: error: PAM: authentication error for illegal user daisuke from 194.143.147.10Mar  6 05:52:24 lara sshd[67198]:...

MongoDB and durability

It looks like my post about MongoDB got a lot more popular than usual, and also provoked a sort-of official response from the MongoDB developer(s). It is fair to metion them together to allow people finding one part of the story to find the other. Since my original post talks about multiple issues and the comments wander through various topics I want to summarize the part of the discussion about durability here. It is...

The "Silent Scene"

Ken Macleod's Night Sessions is one of the most interesting books I've read in a while - I can wholehartedly recommend it to anyone who likes have his/hers imagination be carried away by great (if a bit geeky) ideas. One of the really intriguing examples of this in the book (from a geeky/technical point of view), and also kind of aesthetically pleasing, is the concept of the "silent scene". [I've lent the book to a...

Fun with iSCSI and VMWare

FreeBSD having two iSCSI targets and an initiator, and VMWare in the latest edition having access to the three probably most used remote storage technologies - iSCSI, FC and NFS - there is room to play the combinations and see what is best and most convenient. For start, I wanted to try hosting VMWare images on iSCSI served from a FreeBSD machine. iSCSI is of course "IP SCSI" - a SAN technology that enables...

FreeBSD Forums

Sometime around these days, FreeBSD Forums will celebrate (possibly quietly) two years of its existence! According to its stats, it now has over 10,000 members and over 61,000 posts! Happy second birthday to FreeBSD Forums! All readers should now go and create an account on the Forums if they haven't done it yet. All FreeBSD developers have a default account there, named like their FreeBSD username + "@", e.g. "ivoras@".  ...

Firefox 3.6

OMG! It's been like 15 minutes since Slashdot announced Firefox 3.6 is released and there is still no FreeBSD port for it! What. Are. The. Porters. Doing?!! (I kid, I kid - I saw tantalizing commits coming in :) ) update: it's in the ports now! I want my Personas (aka Google Chrome Skins Rippofs, but also nice looking rippofs) now! #1 Re: Firefox 3.6 Added on 2010-02-12T09:40...

LLVM DevMeeting

Apparently there was a LLVM devmeeting recently. I don't see any FreeBSD-related talks but maybe some day... There some information on LLVM/CLANG progress with FreeBSD on the wiki....

Android SDK on FreeBSD

An interesting thing for Android enthusiasts: there's an Android SDK for FreeBSD 8 available! Didn't try it yet, though :)...

Help test SU+J

Jeff Roberson has published a call for testers for an early development version of the new journal / intent log addition to UFS SoftUpdates code. As previously described, SU+J is the same old SoftUpdates with a small journal added to completely eliminate the need to fsck after unclean shutdown. Judging from Jeff's message, it looks like automated journal recovery is still not turned on....

What's cooking for FreeBSD 9?

I've already started a new page :) Merry hollidays!   (of course, FreeBSD 9's ETA is sometime in 2011 or later)   #1 Re: What's cooking for FreeBSD 9? Added on 2009-12-29T13:27 by Oliver Herold >New driver for AHCI SATA drives   I'm already using this in FBSD8.0R and stable. (ahci_load in loader.conf)   >Solaris-like pwait(1) utility (jilles) (MFCed), refactored rc.d to use pwait(1). pwait is in 8.0 stable, so we will see...

Nicer things in life - VMWare 4, ZFS, virtual hot-plugs

Sometimes, when things go right, they are really beautiful to watch. Like adding a virtual (SCSI) drive to a running virtual machine freshly copied from ESXi 3.5 to vSphere 4, seeing that drive simply appear in the guest system in a hot-plug fashion, creating a ZFS pool on it and watching iostat as data is migrated from the old drive which has gotten to be too small (and uses UFS), to the new one on...

SoftUpdates - the next generation (or SU+J)

Jeff Roberson is working on an addition to UFS softupdates which includes a tiny journal to keep track of things like free space and inode orphaning that were left to background fsck to deal with in the original implementation of softupdates. Basically, it's still the same old softupdates but without the need for (bg)fsck to be run for recovery! The details are in the  linked articles. It appears it will combine the best...

Donate to the FreeBSD foundation!

The FreeBSD foundation is collecting donations for the next year! The Foundation is a regular sponsor of many events, funds trips (including mine) to FreeBSD conferences and occasionally funds projects for various enhancements to FreeBSD. It is an important factor in keeping FreeBSD going! I'd like to rant about what having exposure and good sponsors meens to an open-source project. Specifically, while the FreeBSD Foundation is happy when it gets $300,000 a...

Using Ministat

Recent benchmarks of FreeBSD 8 have provoked a discussion both on benchmark results themselves, on benchmark methodology in general and on upcoming improvements. One of the things mentioned is the small utility used internally in FreeBSD that was recently "blessed" into being a generally available part of the operating system - the ministat. Ministat is a small statistical analisys utility that is mostly used to calculate statistical differences between data sets....

Google Chromium on FreeBSD

I started looking if there is a way to port the Chromium browser (aka the open-source core of the Google Chrome browser) to FreeBSD but it looks like someone has already done it :) Download Chromium for FreeBSD here! The build is created for FreeBSD 7.2 but of course works without problems on FreeBSD 8.0 and probably all higher versions. The consistency FreeBSD maintains in ABI compatibility is really marvelous. The browser build itself looks ok...

How much performance do you lose with VMWare?

I finally have a VMWare ESX 4 (vSphere 4) in production and it's time to see how much we lost this time in the name of greater flexibility. In short: the usual (which is a lot). Here's what unixbench, a mostly single-process benchmark (all except shell scripts) can get on raw dual Xeon 5430 (2.6 GHz, 8 cores total, amd64 in all cases): TEST...

Trying AHCI in 8.0

One of the neat new things in 8 is the native AHCI driver which replaces the old one (which was a part of the "normal" ATA driver set and had not allowed the use of AHCI's full potential). It's remarkably easy to try it in a recent 8.0-RC system. I have set up VirtualBox which has SATA AHCI as an option for emulated disk drives. My machine was created some tiem ago and was configured to...

I'd like my cell phone to replace my laptop!

The Android platform has potential. I've seen the excitement about the new Droid devices and they look good enough to maybe replace my P1i - which I think is one of the greatest mobile phones ever, but only after it's hacked, unlocked and heavily customized because the default firmware is practically castrated by the telcos. The first thing I wanted to know is - is there a ssh client for it. Apparently

A short time with MongoDB

I wanted to try something from the "NOSQL" camp and it looked like MongoDB is one of the darlings of the whole idea / "movement". If this is really representative of the whole group, it's very, very unimpressive. (update: there are better ones ) Update: Here is a funny talk about NOSQL. Update: New post available with updated information. Update: Note that this post was written in 2009. Update: Now, in...

The night of 1000 jails

As FreeBSD 8.0 is right around the corner it's the right time to get it some more exposure. Just for kicks I got the idea to stress the Jails subsystem - the cheap (both in $$$ and resource requirements) OS-level virtualization technology present in FreeBSD for nearly 10 years now. Behold... the bootup of 1,000, count them - 1,000 virtual machines on a single host with 4 GB of RAM. Introduction...

Seasonitis

Apparently, as people in earlier times counted their days in terms of natural events and agricultural work, we still need such divisions of the year but we mostly rely on "thematic" periods of entertainment... It looks like we have: Christmasy - New yearsy period, approximately from the middle of November to the middle of January; a time to get into the chearfull shopping mood and remember why you left the people who you left, buying them appropriate...

Microsoft deserves every bit of negative feelings they get

Microsoft Internet Explorer is currently the ONLY browser not supporting any kind of advanced CANVAS technology. It sucks and if for nothing else, Microsoft deserves eternal damnation for holding back the progress of web technologies. For example (I've just checked), IE 8 is the only browser (the rest are Mozilla Firefox, Opera and Google Chrome) that cannot display this very useful chart library. Among other things that are common but MS refuses to...

VMWare + FreeBSD + ZFS soft-raid with SATA drives - performance

VMWare ESXi is a great free-as-in-beer product bringing professional virtualization within everyone's grasp. Contrasted to VMWare Server 2, I'm yet to find something really serious in ESXi,to rant about, but the lack of software RAID support comes very near. For some strange reason ESXi doesn't support software RAID configurations (officially it's all about "reliability" but I suspect they're either not interested in building management products for sw RAID into the various management "consoles",...

Firefox HTML 5 Ogg Theora playback

Apparently, playing 320x200 15 FPS native HTML5 (Ogg Theora) video in Firefox on FreeBSD takes less than 1% of a CPU core, and that's with X11 compositing enabled (translucent window), but not AIGLX. Nice. However, the "same" video at 640x480 at 30 FPS takes around 10%. It looks like audio is unsynced with video in both cases. Surprisingly, even though the video drivers are presumably vastly better on Windows XP, playing the bigger video takes...

Atom as a web server?

The idea of using a simple, low-profile, low-performance, but also low-cost, low-power consumption and low-maintainance non-mainstream platforms as servers is absolutely nothing new. I have a great fondness for the very idea of using Atom CPUs as servers, and there are already commercial solutions with such a setup available out there. As soon as I can justify using it in a project, I will :) But for now, I have an Atom-based netbook and...

ZFS !experimental

By the judgement of developers, ZFS is no longer considered experimental in FreeBSD :) Thanks to everyone involved, especially Pawel :) ZFS has originally been imported just before 7.0 was released and it had lots of work done on it since then to make it stable enough for general use. It looks as it's finally payed off and there are no more known issues that would make it not recommended for general usage. This "nonexperimental" version of...

UFS file system space allocation policy

Users coming from other systems don't usually know it (and probably don't care, in today's environment of multi-TB drives), but UFS has a rather interesting approach to space allocation which is different from the "usual suspects" like ext2/3, NTFS and FAT(32). Though it sometimes feels "interesting" in the sense of the Chinese proverb, it's time-tested to work. I'm talking about clustering, blocks and fragments. As in more common file systems like ext2/3, NTFS and FAT/FAT32, there is...

FreeBSD as a WiFi Access Point

At a recent Linux users' gathering I temporarily saved the day when a WRT router was practically bricked, by setting up my netbook (Acer Aspire One) running 8-CURRENT as a wireless access point. It had wired connectivity to the Internet from one side and offered WiFi via its Atheros card on the other side. In between it did NAT and protected the LAN side from the Linux hackers, both with ipfw. Here is how I...

How FreeBSD gets released

Believe it or not, FreeBSD as a whole is a really big project. The practice of keeping kernel and userland together in an operating system results in about 45,000 files that comprise the system's source. As with other big projects, some degree of control is needed. (This article is intended for end-users, not developers; developers have a better inside view so they will have to ovelook omitted details) It goes something like this (or at least it...

Save, auto-save or always-save?

This article has some valid points; one of those is: "why do we still hit ctrl-s?" or in more general terms, "why is saving a document a distinct operation?". Since I'm interested in storage it triggered me to some brainstorming on how would something like save-less world work in real life. Firstly, the article is about word processors - WYSIWYG editors for writing (sometimes large) amounts of formatted text, which with graphics and decorations can...

OpenJDK6 port build

I don't know exactly when it happened but there is currently an OpenJDK 6 binary port available for download from the usual place. The reason this is important is because the "official" Diablo/Caffe binaries distributed by the Foundation cannot be legally redistributed by third parties, precluding it for being included in FreeBSD-based products, LiveCD-s, etc. OpenJDK is also free-as-in-speech, which is one of the reasons why it can be (re)distributed. OTOH, the official binaries...

XXTEA for Python

In addition to the moderately well known implementation of Blowfish in pure Python I host, I've released a pure Python implementation of the XXTEA algorithm. It can be downloaded here. The XXTEA algorithm is a bit peculiar for two reasons: firstly, it's tiny when compared to others (see also the original TEA algorithm) and secondly, it has a sort-of inbuilt chaining mode enabling it to process arbitrarily sized blocks (though their...

PHPPgSQL - A PostgreSQL library wrapper for PHP

I'm starting to do a project in PHP and as I usually start from the database definition onwards, I was also considering which approach to use for accessing PostgreSQL. I have never used PDO before so I thought it would be a good chance to learn it and use it. Unfortunately, the very first lines of code I've written with it have shown PDO has a horrible design and fails in its practically every...

I want some convergence here!

It's 2009 damn it, I want some convergence between my "smart" devices! Why can't my photo camera beam photos to my mobile phone? Why can't I edit simple documents on my phone without excessively jumping through hoops? Why can't my car use the GPS from my mobile phone? Why can't I shut down my desktop machine from my mobile phone? Why is even simple communication between my desktop and my mobile phone via Bluetooth so...

The Matrix as an Operating System

I wrote this in 2003. and am republishing it here for fun :) Now I know slightly more and would probably rewrite the whole thing from a more microkernel-oriented angle and with better discussion of security properties but even now, t's a complete overkill. Beware of geeking out! Since one of my fascinations is operating systems' design, implementation and maintenance, ever since I've first seen the Matrix movies I've thought some of the concepts in them can be...

Does YouTube lose data? Does Google?

I've been listening to James Randi semi-regularly and have come across many jevels of information, anecdote, positive thinking and just plain old ranting on favorite subjects. This kind of material is of the type that should be shown at school, regularly, for at least a year. YouTube in general has become a repository not only of college fart jokes but of incredibly useful information in general. It, along with Wikipedia, comprise...

Oracle is buying Sun

Yes, this is old news. I guess the reason I'm writing about it now is that only now the shock of the news is subsiding. Though it's nothing special in the business world, I don't think that casual users of technology have enough appreciation of how monumental event this is in the history of computing and technology-oriented businesses. This truly feels like an end of an era. SUN (Stanford University Network) was one of the...

ZFS v13 in 7-STABLE

ZFS was MFC-ed into 7-STABLE by Kip Macy a short time ago! The differences / additions to the initial import into 8-CURRENT were: MFC ZFS version 13. This includes the changes by pjd (see original messagebelow) as well as the following:- the recurring deadlock was fixed by deferring vinactive to a dedicated thread- zfs boot for all pool types now works Submitted by: dfr

GEOM_LABEL ufsid

One feature in 7.2 that I am guilty of introducing is the support for UFS ID labels in GEOM_LABEL. While it's currently known to annoy some people with messages like GEOM_LABEL: Label for provider da0s1a is ufsid/49d629fac5da1db9. it's actually a feature that will make life easier for many users. The rationale for GEOM_LABEL's existence is to make it possible to reference devices without referencing their physical device nodes. It's an abstraction layer which solves the issue of...

Wolfram Alpha - again the hype is unfounded

It's not that Wolfram-the-person isn't a smart guy, but he could learn a lot from Steve Jobs when it comes to delivering products. Apparently his modus operandi is to create so much hype and incredible claims over whatever he's doing that the end product simply cannot sustain it and then appears to implode. Both A new kind of science and Alpha are good in their own way and have a use but the aura /...

InfraServers

The world seems to be catching on - as opposed to building a large quantity of huge multi-core multi-gigahertz CPUs and using virtualization to manage them, which at best looks too similar to the mainframes from which we have escaped years ago and at worst introduces a point of failure that can bring down many critical services by a single hardware glitch, we now have "infraservers" - a potentially huge number of which can...

PixelCity

If you haven't seen it already (and it's doubtful because it has been making the rounds on the interweblogs), PixelCity is a really nice graphics demo. Some 10 years ago or so I used to pay attention to the new demos that were created and watched them to both relax and admire the technicall skills of their creators. PixelCity renders a procedural city nightscape - a random scattering of buildings, streats and cars looks...

Help test VirtualBox!

The VirtualBox port is progressing nicely. Currently it appears to work better on i386 than on AMD64 but this will probably be fixed soon. The developers have issued a call for testers so all interested users should go and help! I've written about VirtualBox earlier. #1 Re: Help test VirtualBox! Added on 2009-05-20T23:49 by Sergio Lima I perform up2date my laptop system (current: FreeBSD varyamo.mindshining.org 8.0-CURRENT FreeBSD...

More fixes to ZFS

Excellent news for ZFS users - Kip Macy has just committed some long-discussed patches to ZFS to increase its stability. Most paches have some equivalent in OpenSolaris, and they should bring FreeBSD more inline with performance and stability expected of ZFS in Solaris. All users of ZFS are invited to test it (again). The patches clean up some aspects of the ARC and ZFSs interfaces to FreeBSD. It should greatly help with running out of kmem...

VirtualBox on FreeBSD

VirtualBox has recently made significant progress on FreeBSD! Apparently, they are pretty close to having a full working version on FreeBSD as a host. Here are some mailing list references: http://permalink.gmane.org/gmane.comp.emulators.virtualbox.devel/1370 http://permalink.gmane.org/gmane.comp.emulators.virtualbox.devel/1380 http://permalink.gmane.org/gmane.comp.emulators.virtualbox.devel/1329 I think this development is on par or even more important then ZFS, as FreeBSD currently has no full virtualization options (as a host) other than QEmu.   #1 Re: VirtualBox on FreeBSD Added on...

Rise of "low-state zone" games

"Zoning in" is something most people are familiar with, and geeks even more so. The topic / environment / goal for zoning in can be practically anything that's perceived as interesting enough to occupy the brain enough to ignore most external stimulus - from physically working with stuff - for geeks this usually means modelling, electronics work and other "building stuff" activities - to mental activities - programming is common here. Games are of course...

Some virtualization benchmarks

I did some benchmarks of the three currently most prominent virtualization platforms: VMWare ESX 3.5 U3, Citrix XenServer 5.0 U2, Microsoft Hyper-V 2008 R2 on three 64-bit guest operating systems: Ubuntu 8.10, FreeBSD 7.1 and Windows Server 2008 R2 beta. The results are mostly better then I thought they will be. Especially suprising was FreeBSD's more than decent performance which actually lead the others in one benchmark. The benchmark was serving PHP pages with Apache, with...

mdcached

For some time now I have been working on and off on a project similar to memcached. I'm using the project also for academic purposes, writing papers on what I've done with it, etc. and maybe it's time to give it a little bit more of exposure. The project is mdcached, also available on Sourceforge.net. The intent for it is to be a high-performance cache engine similar to memcached with some extra features built-in (mostly,...

Good experiences with X.Org autoconfiguration

There has been some grumbling about the new X.Org autoconfiguration and its dependancy on hald, but I've actually had decent experience with it. In short - it's so automated it's boring. I have no xorg.conf on the AMD64 system I'm typing this post from. Simply none at all. X.Org and Xfce 4.6 were installed from port, startx started and I have a nice desktop. My video card is powered by the radeonhd driver, my mouse is...

SSL optimization and security

I came across this article about home-made HTTPS accelerator systems (basically a HTTPS proxy to be put in front of proper servers), and one important thing seems to be ignored - the encryption algorithm. edit: this post was updated in 2012. I don't have servers that require high TPS rates over HTTPS (we use it mostly for  logins and administrative tasks, regular users don't need it), so I usually just use the default settings that come...

Java on FreeBSD

For some reason, some people are still reluctant to use Java on FreeBSD. Though it had a relatively bad history with regards to its licencing and cannot be installed automagically, the official binary packages have passed Sun's tests (this is important!) and are as stable as they should be. The procedure to install Java is: Install java/javavmwrapper Go to http://www.freebsdfoundation.org/downloads/java.shtml and download the appropriate package Install the package The package itself depends on a small number...

Fuckup fairy narrowly avoided

The default GEOM partition slicer in FreeBSD has been changed relatively recently - now it's the unified slicer called GPART. It supports many different partition types with a consistent interface. Unfortunately, it isn't 100% compatible with previous slicers - devices can get renamed! Even though I knew this in principle, I still managed to create a ZFS file system that clobbered a partition table I really wished I didn't clobber. In retrospection it probably wasn't as...

Magical incantations for Samba

Judging from the frequency of discussions on the mailing lists, Samba is notoriously difficult to tune for FreeBSD. I decided to play a little on a platform where every bit of performance helps, an embedded Geode-based computer, with some good results. Before tuning, I was able to achieve 6-7 MB/s in either direction. I've then upgraded from default samba to samba 3.2 (newsflash: samba33 is in the ports!) and after some trial and error fiddling with...

What happened to finstall?

I occasionally get questions about the finstall project - more or less to the not of "what happened to it?". It's not very hard to conclude that its development has been stalled. I'm personally sorry about that and would very much like to continue working on it, but as always, real life got in the way. I still don't consider the project abandoned, and occasionally when I have the time add something to it....

Optimizing for Atom

Since the Atom CPU is understandably (and expectedly) slow, it's obviously interesting to attempt to find the perfect compiler optimization, the magic bullet that will bring out the best from the applications. One common advice, easily found with Google, is to treat it as if it were pentium4. This can be theoretically explained by the CPU having HTT (not that it's important for single-threaded optimizations), or that it's microinstructions are simpler, trading high clock rates...

Installing FreeBSD on Acer Aspire One netbook

With the goal of losing too much time for questionable gain, I decided to install FreeBSD as a multi-boot option on my Acer Aspire One netbook. It did not go smoothly and I lost much more time than I hoped I would but I now have a FreeBSD netbook to play with :) After trying other things, I resorted to using UNetbootin, a very handy utility that prepares bootable USB drives with pre-defined...

Google's own DoS on the Internet

It's well known and propagated that in the last hour or so Google has been flagging everything as a potential security risk / malware carrying site. I won't go into philosphy of the whole thing, just want to share an observation: I've noticed it myself but on a result from Wikipedia and I just ignored it - "no way is Wikipedia spreading malware". Only later I confirmed  that it's Google's problem (on other forums,...

ext4 looks good

Early adopters are starting to write about the ext4 file system and it looks good. In this writeup the re-read anomaly is probably due either to invalid benchmarks or to hacks to page replacement LRU in page cache. Mixed workload and random write are significant - with random write being close to 10% better on ext4. Is anyone interested in porting it to FreeBSD? :) #1 Re: ext4...

Where *does* the mass of trees come from?

Some months ago I watched a TED talk given by Jonathan Drori in which he asks some interesting and seemingly tricky questions. The questions are meant to test general scientific knowledge in (somewhat) non-obvious situations. I was right about all of the questions he gave, except for "where does the mass of the tree come from", which I couldn't really decide on the spot. The talk has itself been talked about and you...

FreeBSD 7.1

The long expected new version has just been released! Why should you care? If you remember how awesome the 4.x branch was, you'll know what I mean when I say 7.1 is approaching that aweseomness. Not that the old versions were garbage, but if 7.1 is 0.98 on the awsome scale, 7.0 is 0.95, 6.x is something like 0.75 and 5.x is better not mentioned. FreeBSD 7.1 updates the technologies present in 7.0 and brings previews of...

AIX lpars and wpars... sounds familiar

I'm reading this article about LPARs and WPARs in AIX and it sounds familiar. It looks like LPARs are equivalent to full virtual machines, like in VMWare, VirtualBox, or Xen, and WPARs look like Jails. When Xen gets ready, it will be possible to implement this functionality - practically all of it - in FreeBSD. Jails have been implemented for 10 years+. #1 Re: AIX lpars and...

Donate to the FreeBSD Foundation

Like in most ventures, there are basically two ways things get done in FreeBSD - either people do it themselves (mostly) or they pay someone else to do it. By donating to the FreeBSD Foundation you can help develop FreeBSD without needing or knowing how to code. The Foundation is very close to its goal for this year (which is surprising because of the overall apparent shortage of money) and needs only a bit more...

The circle of technology and FreeBSD

I've been reading the FreeBSD Release Notes for version 1.1, published in 1993. Two things are interesting to me here: The only name in the old core team I recognize is Jordan Hubbard, and he's not a FreeBSD developer any more (joined Apple for Mac OS X) Some problems, both technical and organizational, are always repeating themselves. These things are "normal" and present not only in Open-source projects but in the entire industry (and of course in...

Virtual this and hyper that

One of the most important aspect of "full" virtualization products, at least for me, is the ability to create and deploy "appliance"-like virtual machines with server software (hmm, like PostgreSQL) packaged together with an operating system as a black box. This package is then independant of the underlying OS so today I can deploy it on Windows, tomorrow on Linux - and maybe one day if host virtualization solutions get developed - on FreeBSD. A...

Freenet6

I was talking with a friend about IPv6 and I remembered hearing about public IPv6 gateways. 6bone is gone but go6 (or is it freenet6?) is apparently its public successor. It's really so extremely simple to set up I was actually kind of dissapointed at having nothing intelligent to do except simple config file editing. ursaminor:~> ping6 www.freebsd.orgPING6(56=40+8+8 bytes) 2001:5c0:8fff:fffe::d019 --> 2001:4f8:fff6::2116 bytes from 2001:4f8:fff6::21, icmp_seq=0 hlim=55 time=218.891 ms16 bytes from 2001:4f8:fff6::21, icmp_seq=1...

My wish list for FreeBSD

In a perfect world, where I somehow manage to have millions of $currency, I'd sponsor the development of: Host-side virtualization for FreeBSD: either qemu / kvm, virtualbox, vmware or xen-dom0. This is the first and "hot" item on this list - there are many opportunities where I could use FreeBSD as a host for other operating systems instead of reverse (I don't care for "guest" support like xen dom-U since full virtualization solution exist and are...

Page 56

As seen elsewhere:   Grab the nearest book. Open it to page 56. Find the fifth sentence. Post the text of the sentence in your journal along with these instructions. Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST. What I have is: A soft blue twilight lay across the city, increasing by a hundredfold its air of magic and mystery. -- from Roger Zelazny's & Robert Sheckley's "A Farce to...

Aaaaand it's there! The Big ZFS Update!

And there it is - ZFS update for FreeBSD containing a huge amount of changes! Pawel has commited the hugely awaited update to ZFS in -CURRENT. It's time to try ZFS again! Big thanks to Pawel and other contributors! List of changes from Pawel's commit message:  Update ZFS from version 6 to 13 and bring some FreeBSD-specific changes.  This bring huge amount...

"Big ZFS update" expected? :)

I've just seen a commit message from Pawel: Author: pjdDate: Mon Nov 17 20:11:53 2008New Revision: 185027URL: http://svn.freebsd.org/changeset/base/185027Log: Increase the limit for the Big ZFS Update.Modified: svnadmin/conf/sizelimit.confModified: svnadmin/conf/sizelimit.conf==============================================================================--- svnadmin/conf/sizelimit.conf Mon Nov 17 19:00:36 2008 (r185026)+++ svnadmin/conf/sizelimit.conf Mon Nov 17 20:11:53 2008...

Linux adventures

In the last two days I had two interesting experiences with Linux. The first was with Ubuntu upgrade on Acer Aspire One from 8.04 to 8.10, which was abruptly interrupted but managed to recover extremely well, and the second with RHEL 4.something where I decided using pkgsrc would be much saner than dealing with their native packages (or more correctly, the lack of packages). Ubuntu story: I carried the gadget to work...

Even Google can fail user interface

Is there no vendor today than can actually pull off a an application version increase without a serious regression in features or usability? As suggested by an in-house ad on Gmail, I've installed a new version of Mobile Gmail, an application written in Java (J2ME) that provieds a full user interface to access Gmail messages, reply to them, etc. I was very happily using version 1.0 of this application which, though it had some annoyances,...

Beware of stupidly configured ADSL routers

My old ADSL modem+router was acting strangely (it wasn't 100% stable even at the start - but it was free...) so I bought a new one. Good thing I, practically by accident, tried to connect to it from the outside (from the WAN / Internet side) via Telnet. There it was, wide open, with a default username/password of "admin" / "admin". Microtek's ADSL modem+router apparently has this interesting feature built-in by default: it's wide open...

Vista out of control

I have a laptop which came with Windows Vista by default. Though I immediately set up dual-boot with Ubuntu, I still need Windows for some applications. The resource consumption by Vista is simply amazing. I've read the rants and the blogs but it's a different feeling when you see it for yourself. Vista, with only the Task Manager application running, takes nearly 1 GB of memory and 7% of a dual-core CPU (i.e. 14% of...

cp(1) and cat(1) in the Century of the Fruitbat

FreeBSD has frugal ancestors, created when memory was at premium. The code that was once optimal for the hardware, even using heuristics to adapt to various circumstances, might be underperforming now. An excellent example for this was the transition from a system highly optimized for single-processor hardware to multi-processor (SMP) hardware, which is only recently being completed. In a much simple case, cp(1) and cat(1) were using buffers of 64 kB (and sometimes even 1...

VMWare, SMP and VMI

VMWare Server 2 has two significant features that look very good in theory: virtual SMP support (vSMP) and VMI. Virtual SMP is just what it seems to be - support for multi-CPU virtual machines for SMP operating systems. VMWare Server VMI is Virtual Machine Interface, a paravirtualization interface (a kind of API, actually) that guest operating systems can use to make virtualization easier and more efficient for the host (similar to Xen). I've tested...

pbzip2 1.0.3beta2

Jeff Gilchrist has released a new beta of pbzip2 - the parallel version of bzip2 compressor, usable on SMP machines. It has fixed one important error from beta1 (that actually came from me) and should be stable and useful for general purposes. This is a call for testers :) If you need the increased performance it achieves on multi-core and multi-CPU systems, please try it out. This version is good enough to be considered a...

Countries I visited

I've found Big Huge Labs Map Maker - it uses Google Maps maps to easy colorize countries on a map. A map of countries I visited. It's not much but I hope to make it more colorful soon :) Make yours @ BigHugeLabs.com #1 come to japan :) Added on 2008-10-23T12:00 by dai We'll open AsiaBSDCon...

ULE topology dump

I'm working (instructed by Jeff Roberson) on a small patch that exposes the CPU topology used by ULE as a read-only sysctl variable. Here's an example output: kern.sched.topology_spec: 0, 1, 2, 3, 4, 5, 6, 7 0, 1, 2, 3 4, 5, 6, 7

pbzip2 and tar

I have several large nightly backups that run almost the whole night and was searching for ways to improve both the speed and the archive size. The difference between bzip2 and gzip is large enough to use bzip2 and fortunately there's something that can improve its horrible performance: pbzip2. I also thought that sorting the files by their extension, like Windows compressors do, will reduce the final archive size, but this didn't go very...

Addition to FreeBSD's package infrastructure: pkg_trans

Some time ago I've described a feature I wished for the ports & package infrastructure, and now I have a working prototype, at least for some of it. The main idea is to enable grouping of package operations into "transactions" that can be undoed as a whole. My own examples when this would be needed are Install some large group of packages, like KDE or GNOME. Don't like it, want to delete all packages installed during...

Book review: Network administration with FreeBSD 7

Some time ago I was asked to do a review of a new book, Network administration with FreeBSD 7. As far as I can tell this is a first FreeBSD-themed book from its publisher (Packt publishing) which is specialized in IT-related titles - and it is a good start! The book has a cookbook approach to a wide range of administration topics with FreeBSD. It is based on the most important FreeBSD release...

QED Conference, Rovinj

Quality in Enterprise Development (QED) is a conference organized by CROZ to bring together people involved in software development, especially on a higher level. It's a small conference, and a sort-of continuation of the last-year's Project&Portfolio Management (obviously directed at a different level), but its exceedengly good organization and careful selection of speakers make it one of my favorite small conferences. It was held in the beautiful town of Rovinj which had enough sun...

Mass Effect

I'm not a heavy game player but I carefuly watch everything that comes out of BioWare. Though not everything they did recently was as good as Baldurs Gate and others from the era, Mass Effect really comes close. Cue bombastic announcer voice: RPG ... IN ... SPAAAACE! I like SF-based settings more than the usual high fantasy "elves with big swords and even bigger ears" type so Mass Effect had a good chance with me even before...

Trip to Montpellier

I've been to Montpellier! The official purpose was to attend IBM's storage conference, but the whole place and the company I had were much nicer than the conference itself. It's a very Mediterranean place with some stunning architecture and a very nice blend between old styles of buildings and recent, modern styles. I didn't get the chance to visit one of its beaches during daytime but from what I've seen they should be as good...

Acer Aspire One

I'm travelling relativly often so I bought myself an Acer Aspire One to avoid lugging my main laptop around when I don't need it. So far, it's been suprisingly good. I can usually find a flaw everywhere but this is a suprisingly good piece of equipment. It has an Intel's Atom CPU running at 1.6 GHz (slowing down to 800 MHz in power-save...

VMWare Server 2 performance

It looks like VMWare Server 2 is right around the corner. Despite that I still don't like it's new web user interface, the actual virtualization product is very good. Here's something about VMWare Server 2's performance with FreeBSD. I've tested 8-CURRENT with debugging disabled, on VMWare Server RC2. The tests were done on exactly the same installation, with VMWare using it as a "raw" device. File system parts of the benchmark should be ignored...

Versioning configuration files with Mercurial

One common administrative problem is how to protect from misconfiguration, or at least have a source of known-good configuration to use when the production breaks. One possible solution for this is to put all configuration files (in /etc) in a VCS like CVS. Of course CVS is not the only one... By "configuration files" I mean "the contents of /etc and /usr/local/etc". Since I've been in the situation where I'd very much like to know how...

Tincture

In my first post I said I'll describe Tincture, the web application behind this web site. Read it here after the "continue reading" link. Tincture is a blogging application intended to be simple to use and have really high performance. The former was accomplished by having WYSIWYG editors and logically presented options and the latter by having it generate plain HTTP files on the server. Users visiting this blog are actually accessing plain files - there...

Ubuntu and Vista

I got a new laptop and it comes with Vista preinstalled. Since I actually do useful work with Windows (and Linux and FreeBSD) I want to keep it there for now (later it might get overwritten with XP) and install Ubuntu as the second option... It's a HP laptop and by default it created three primary partitions (bad HP!) The first one is Windows' default partition taking up around 95% of disk size and the rest...

New web URL-s for common FreeBSD resources

Wolfram Schneider has added new convenient URL redirections on www.freebsd.org that are easy to understand and point to complex but much used URL-s. Continue reading this post to find out about them. Here are the new URL-s: http://ports.freebsd.org now points to http://www.freebsd.org/ports/ http://man.freebsd.org/fdisk now points to http://www.freebsd.org/cgi/man.cgi?fdisk A variation on the above, using section IDs: http://man.freebsd.org/fdisk/8 These shortcut URLs are more logical and easier to use than the full versions....

Impact of malloc debugging in CURRENT

Before a bunch of CURRENT FreeBSD code is blessed into a STABLE branch, it has many debug options enabled that help developers track and solve problems. The most well known options are WITNESS and INVARIANTS in the kernel, and MALLOC_PRODUCTION in the userland (libc). Each of these has a significant impact on the system's performance if configured for debugging. WITNESS and INVARIANTS usually have the biggest impact on performance as they are directly influencing core functionality...

VMWare Server 2 redux

Recently I had trouble running VMWare Server 2 (RC1), so much I gave up on it. I tried again, this time RC2 on Linux and the experience is a little better - at least it works. I had to use tricks to force it to use a raw (physical) drive (this feature has been officialy removed) and a different emulated ethernet device. I wanted to boot my raw partition with FreeBSD under VMWare Server 2...

New USB stack coming to CURRENT

The new USB stack (formerly known as HPS-USB) will soon be committed to current. Some of the new features are: - Full support for Split transactions, which means you can use your full speed USB audio on a high speed USB HUB. - Full support for HS ISOC transactions, which makes writing drivers for various HS webcams possible, for example - Full support for USB on embedded platforms,...

Status of superpages support

Superpages are mostly done in -CURRENT and everyone who wants to test them is welcome. There are a few caveats; interested users should read this post by the author of superpages implementation. This posting is quoted and annotated in the second part of this article. Superpages are not enabled by default but support for them is a standardpart of the amd64 and i386 kernel on HEAD. You can enable superpages oneither of...

Undelete files on FreeBSD?

The short answer is: you can't. The long answer is that it might be possible to recover some lost data, with quite good chances, but absolutely no guarantees. See the rest of the post for details. I've set up a 8-CURRENT machine in VirtualBox and when I needed a 8-CURRENT system to start working on pkg_trans, I was too lazy to set up another one and started using the VirtualBox VM. As I don't quite trust...

Xen is coming to -CURRENT

It looks like Xen is finally coming to FreeBSD. It's still under development and not yet ready for testing by a wider audience. AFAIK this is only client ("domU") support, host support is still a long way off. If you've been living under a rock, Xen is a very popular open source virtualizaion framework. Traditionally it's used with paravirtualized kernels (i.e. kernels must be modified before they can host and run Xen VMs) but...

Bank card security problems

There was another big case of circumventing the security of various cards by skimming... There's one thing that's always puzzled me: why not integrate the PIN input buttons on the card? It's not like we can't create flat electronic contact buttons that can be built into the card (e.g. low precision touch panels - they're cheap since they don't have to be transparent; I imagine something like this), and the processing isn't that demanding....

Geode LX support in FreeBSD

A while ago I was dissapointed with missing support for Geode CPUs in FreeBSD. This is rapidly changing in the last few months. At the time I first tried it, there were somewhat serious problems in getting the Fit-PC I acquired (and still use, to my satisfaction) with FreeBSD: The boot loader didn't work (or at least failed to boot from the install CD) Cryptography additions to the CPU were not supported USB was restricted to USB...

I'd buy this

BoingBoing has a what-if article about a portable computer / tablet / cell-phone that looks extremely nice. Update: yea, verily not a year has passed and a product not entirely dissimilar to this one is available. Still lacks the cool Blade Runner screen / background. The way I imagine it, the screen either slides or opens to reveal the keyboard below it, which is then approximately...

Hello, world!

Welcome to the relaunch of my personal blog. I've previously written stuff either on the FreeBSD blogs system or on my own general web site. As I want to expand the topics on which I write and reduce administrative overhead, this is to become my "official" place to write things. I'm also breaking in a new blogging application and it will take some time to solve all the little problems. Hopefully, everything will work ok in a...