archive-ca.com » CA » E » EVANJONES.CA

Total: 397

Choose link from "Titles, links and description words view":

Or switch to "Titles and links view".
  • The Four Month Bug: JVM statistics cause garbage collection pauses (evanjones.ca)
    multiple file systems ext2 ext3 ext4 btrfs extremely common and xfs less common The pauses occur even if the I O is to a separate disk or if you call mlock The only workaround I ve found is to place the mmap ed file in tmpfs a RAM disk or disable it completely I don t know exactly why the Linux kernel does this but the pauses do not seem to occur for reads so the Linux kernel is marking these pages read only A friend suggested this is the kernel s way of reducing the write I O rate under overloaded conditions If you know exactly why the kernel is doing this I would love to hear it Since I ve discovered this a number of services at Twitter are experimenting with adding the XX PerfDisableSharedMem JVM flag and seeing a significant reduction in worst case JVM pause latency I highly recommend testing this out if you run a latency sensitive service Symptoms I first noticed some JVM garbage collection pauses that were taking more real time than user plus system time Here is an example garbage collection log entry 2014 12 10T12 38 44 419 0000 58758 830 GC Allocation Failure ParNew 11868438K 103534K 13212096K 0 7651580 secs 12506389K 741669K 17406400K 0 7652510 secs Times user 0 36 sys 0 01 real 0 77 secs The last section of this line shows that this garbage collection took 770 milliseconds of real time real 0 77 However the garbage collection threads only consumed 370 milliseconds of CPU time user 0 36 sys 0 01 The JVM uses multiple threads for GC so normally the user plus system time is multiple times higher than the real time This suggests the kernel has blocked the GC threads for some reason The

    Original URL path: http://www.evanjones.ca/jvm-mmap-pause.html (2016-04-30)
    Open archived version from archive


  • Finding the Four Month Bug: A debugging story (evanjones.ca)
    that we don t allocate more CPU than we actually have and that even in the worst case other services should only be able to steal the CPUs for tens of milliseconds not hundreds of milliseconds It could still be possible for processes that are running outside the containers to cause this I spent a lot of time digging through log files to try and find some tasks that might be running while these suspicious pauses occurred I did find one or two instances where an expensive task seemed to be running at approximately the right time but never anything that could explain all the pauses Investigating this theory was mostly a waste of time I did learn about a number of system management and monitoring tasks that we run but otherwise this was a complete waste of time Safepoints Possibly affected Garbage collection is the most well known reason the JVM pauses an application but there are many others such as re compiling a method Thanks to Carsten s suggestion I started looking at safepoint logs and noticed that on instance that have suspicious garbage collection pauses there were usually safepoint pauses that also took hundreds of milliseconds with no clear pattern or cause Unfortunately the JVM does not record the real user system times for safepoints so while I suspected that these pauses were related but wasn t 100 sure Related problems The Internet contains many discussions about interactions between the JVM and the Linux kernel like page scan attacks zone reclaim transparent huge page compaction and splitting and that writes to cached files can block I was very excited after I read about blocking cached writes If GC threads write to a log using the write system call that could occasionally block for hundreds of milliseconds This would cause the garbage collection to take a long time but have low user and system time since the process is not running A brief look at the Open JDK code seemed to suggest that garbage collection logs are written by the GC threads themselves I wrote a small C program and was able to reproduce pauses on a clean virtual machine However when I asked Carsten about it he told me that Twitter s JDK has a custom logging patch that writes to a circular buffer and does the writes on an unrelated background thread This means this couldn t be our problem but I would love to hear if this can happen with OpenJDK A critical correlation One day I noticed an instance of a service that had these pauses far more frequently than others many times each day I decided to graph nearly every system level statistic I could since I figured there must be something different about this system or process After an hour or two I finally saw a suspicious correlation with the milliseconds spent writing exported by proc diskstats on Linux and displayed with vmstat D This shows that in many cases when the

    Original URL path: http://www.evanjones.ca/jvm-mmap-pause-finding.html (2016-04-30)
    Open archived version from archive

  • How to Use UTF-8 with Python (evanjones.ca)
    use the generic string base class basestring if isinstance s basestring True for both Unicode and byte strings Reading UTF 8 Files You can manually convert strings that you read from files however there is an easier way import codecs fileObj codecs open someFile r utf 8 u fileObj read Returns a Unicode string from the UTF 8 bytes in the file The codecs module will take care of all the conversions for you You can also open a file for writing and it will convert the Unicode strings you pass in to write into whatever encoding you have chosen However take a look at the note below about the byte order marker BOM Working with XML and minidom I use the minidom module for my XML needs mostly because I am familiar with it Unfortunately it only handles byte strings so you need to encode your Unicode strings before passing them to minidom functions For example import xml dom minidom xmlData u français Comment ça va Très bien français dom xml dom minidom parseString xmlData The last line raises an exception UnicodeEncodeError ascii codec can t encode character ue7 in position 5 ordinal not in range 128 To work around this error encode the Unicode string into the appropriate format before passing it to minidom like this import xml dom minidom xmlData u français Comment ça va Très bien français dom xml dom minidom parseString xmlData encode utf 8 Minidom can handle any format of byte string such as Latin 1 or UTF 16 However it will only work reliably if the XML document has an encoding declaration eg xml version 1 0 encoding Latin 1 If the encoding declaration is missing minidom assumes that it is UTF 8 In is a good habit to include an encoding declaration on all your XML documents in order to guarantee compatability on all systems When you get XML out of minidom by calling dom toxml or dom toprettyxml minidom returns a Unicode string You can also pass in an additional encoding utf 8 parameter to get an encoded byte string perfect for writing out to a file The Byte Order Marker BOM UTF 8 files sometimes start with a byte order marker BOM to indicate that they are encoded in UTF 8 This is commonly used on Windows On Mac OS X applications eg TextEdit ignore the BOM and remove it if the file is saved again The W3C HTML Validator warns that older applications may not be able to handle the BOM Unicode effectively ignores the marker so it should not matter when reading the file You may wish to add this to the beginning of your files to determine if they are encoded in ASCII or UTF 8 The codecs module provides the constant for you to do this out file someFile w out write codecs BOM UTF8 out write unicodeString encode utf 8 out close You need to be careful when using the BOM and UTF 8 Frankly

    Original URL path: http://www.evanjones.ca/python-utf8.html (2016-04-30)
    Open archived version from archive



  • Using Unicode in C/C++ (evanjones.ca)
    that belief into your software particularly as computers spread through more of the world and as people create new characters The bad news is that converting to and from these wchar t values involves C s complex locale library I understand that this locale stuff is complicated because internationalization is hard but I have never figured this API out On my system calling setlocale LC CTYPE en ca UTF 8 enabled UTF 8 output although there probably is a better way to do it If you need to do conversions to and from specific encodings the minimal approach is to use iconv which is part of the C library If your system doesn t supply an implementation libiconv is released under the LGPL licence allowing it to be used with commercial as well as open source code If you need to manipulate UTF 8 strings you may want to consider GLib which is under the LGPL licence and includes many helpful UTF 8 string routines Finally if you need to do any complicated text manipulation ICU International Components for Unicode is the way to go It can do location and language specific tasks such as formatting dates and times uppercasing and lowercasing strings and alphabetically sorting text There are two questions when writing software that must deal with Unicode What format do you use for data that goes in and out of your software and what format do you use internally For the internal format as Tim Bray explains pick UTF 8 or UTF 16 and stick with it It hardly matters which one as long as you are consistent However it seems to me that UTF 8 is the standard choice of most software I encounter these days so that is what I personally recommend For external data things

    Original URL path: http://www.evanjones.ca/unicode-in-c.html (2016-04-30)
    Open archived version from archive

  • Implementing a Thread Library on Linux (evanjones.ca)
    and in general signal handlers There is an excellent post from Linus Torvalds to the Linux Kernel Mailing List that explains the advantages of this implementation This means that in the process list in Figure 1 the processes may actually be threads For example the four mozilla bin processes are actually threads inside a single process On Linux kernel threads are created with the clone system call This system call is similar to fork in that it creates a task which is executing the current program However it differs in that clone specifies which resources should be shared To create a thread we call clone to create a task which shares as much as possible The memory space file descriptors and signal handlers The signal to be sent when the thread exists is SIGCHLD so wait will return when a thread exits The first challenge is allocating a stack for the thread The simplest solution used in libfiber is allocating the stack on the heap using malloc This means that an estimate of the maximum stack size must be made If the stack grows larger memory corruption will occur A solution used by bb threads is to create a barrier at the bottom of the stack with mprotect This way the thread will cause a segmentation fault when it overflows the stack The best solution used by the Linux pthreads implementation is to use mmap to allocate memory with flags specifying a region of memory which is allocated as it is used This way memory is allocated for the stack as it is needed and a segmentation violation will occur if the system is unable to allocate additional memory The stack pointer passed to clone must reference the top of the chunk of memory since on most processors the stack grows down This is done by adding the size of the region to the pointer returned by malloc To avoid a memory leak the stack must be freed once the thread has exited The libfiber library waits for threads to exit using wait then frees the stack using free Figure 3 shows an example of creating a thread See libfiber clone c in the libfiber package for the full implementation For details about how Linux s pthread implementation creates threads see The Fibers of Threads from Linux Magazine include malloc h include sys types h include sys wait h include signal h include sched h include stdio h 64kB stack define FIBER STACK 1024 64 The child thread will execute this function int threadFunction void argument printf child thread exiting n return 0 int main void stack pid t pid Allocate the stack stack malloc FIBER STACK if stack 0 perror malloc could not allocate stack exit 1 printf Creating child thread n Call the clone system call to create the child thread pid clone threadFunction char stack FIBER STACK SIGCHLD CLONE FS CLONE FILES CLONE SIGHAND CLONE VM 0 if pid 1 perror clone exit 2 Wait for the

    Original URL path: http://www.evanjones.ca/software/threading.html (2016-04-30)
    Open archived version from archive

  • Getting Started With ns2 (evanjones.ca)
    the S5 tools about how to use ns2 for 802 11 simulations This steps through a simple example scenario and uses my tools to process the output Tools for ns2 A collection of scripts for processing ns2 trace files and plotting the output Some are command line tools others are modules that are usable from Python scripts If you have any other tools which might be useful please let me

    Original URL path: http://www.evanjones.ca/ns2.html (2016-04-30)
    Open archived version from archive

  • Efficient Java I/O: byte[], ByteBuffers, and OutputStreams (evanjones.ca)
    sun nio ch IOUtil write which checks the type of ByteBuffer If it is a heap buffer a temporary direct ByteBuffer is allocated from a pool and the data is copied using ByteBuffer put The direct ByteBuffer is eventually written by calling sun nio ch FileDispatcherImpl write0 a JNI method The Unix implementation finally calls write with the raw address from the direct ByteBuffer Microbenchmarks To quantify the performance differences I wrote a small microbenchmark that compares byte direct ByteBuffers and heap ByteBuffers The three tests are filling the buffers with integers filling them one byte at a time and writing them to dev null When writing to dev null the operating system does no work so this should measure best case Java performance I ran the tests on a 2 53 GHz Intel Xeon E5540 Core i7 Nehalem architecture Linux system with Sun s Java 1 6 0 16 I also tried a JDK7 beta 1 7 0 ea b74 and the results were the same although jdk7 s numbers were slightly better The graphs below show the results and the source code is linked at the bottom The interesting observations OutputStream When writing byte arrays larger than 8192 bytes performance takes a hit Read write in chunks 8192 bytes ByteBuffer direct ByteBuffers are faster than heap buffers for filling with bytes and integers However array copies are faster with heap ByteBuffers results not shown here Allocation and deallocation is apparently more expensive for direct ByteBuffers as well Little endian or big endian Doesn t matter for byte but little endian is faster for putting ints in ByteBuffers on a little endian machine ByteBuffer versus byte ByteBuffers are faster for I O but worse for filling with data Source code javanetperf tar bz2 includes previous benchmarks Conclusions Direct ByteBuffers

    Original URL path: http://www.evanjones.ca/software/java-bytebuffers.html (2016-04-30)
    Open archived version from archive

  • Farewell to MIT (evanjones.ca)
    that I don t personally find writing papers to be very fulfilling and I d rather make things that people will actually use I m trying to do that now I also learned how to do academic research While my publication record is not stellar it is also not terrible Many people have written better graduate school advice than I could so I won t try However there is one important piece of advice I wish I could give to my past self when I started my PhD Actively collaborate with others By active I mean work on the same code the same experiments and the same papers with someone else Discussing ideas is a first step but it is not the same thing In my experience most graduate students at least in computer science work alone I understand why because I did this myself Working with others is difficult You end up wasting time explaining things to others having meetings and writing emails and trying figuring out how you are supposed to use their work Even worse someone may not think your idea is interesting enough to work on These challenges certainly made me go off into my own little corner and work by myself because I was working on something interesting damnit I didn t have time for dumb meetings However the most rewarding experiences I had were when I was actively collaborating with others on the H Store and Relational Cloud projects We achieved more than I could have by myself I learned more than I would have by myself and most importantly it is more fun to work with others You have people to commiserate with when papers get rejected celebrate with when you submit them and to help you with tough problems Even those wasteful

    Original URL path: http://www.evanjones.ca/farewell-mit.html (2016-04-30)
    Open archived version from archive