From: cr88192@gmail.com   
      
   On 3/21/2023 7:14 PM, Dan Cross wrote:   
   > So, is anyone here actually working on developing real, novel   
   > operating systems on modern hardware here?  It would be   
   > interesting to have a USENET outlet for such things, which I had   
   > hoped this newsgroup might be, but it seems dominated by other   
   > things.   
   >   
      
   I have developed an OS of sorts ("TestKern") for a custom CPU ISA (BJX2)   
   initially mostly because it seemed like too much effort to try to port a   
   "real" OS (such as Linux or BSD); and ended up going in different   
   directions on a few fronts.   
      
   Some aspects of its design resemble a sort of hybrid of Linux like and   
   Windows like design choices, with some "twists".   
      
   Still pretty rough, crude CLI only, still no GUI or similar (had   
   experimented with windowed redraw, but its performance fell short of   
   "usable").   
      
      
   Binary Format:   
      PEL4, essentially LZ4 compressed PE/COFF;   
      Loader will also accept regular PE/COFF;   
      Theoretically, an ELF loader also exists, but is mostly untested.   
      
   The ABI does tweak some things slightly, as the PE/COFF image is   
   interpreted as being two major parts:   
      A read-only area containing ".text" and other read-only sections;   
      A read/write are containing ".data" and ".bss" and similar;   
      These may be put into different areas in the address space;   
        Depends mostly on the discretion of the loader;   
        The kernel image is loaded as a single monolithic part.   
      
   Some other sections and structures differ from those in "mainline"   
   PE/COFF variants. Similarly, PEL4 binaries omit the MZ stub and header   
   (and start with a PEL4 magic followed directly by the COFF headers; with   
   the headers uncompressed but all the section contents are LZ4 compressed).   
      
   I tested a few compression schemes, but LZ4 seemed to work out as best   
   "on average" in this use-case (I have another compression scheme which   
   works better on average for general data; but tended to do worse than   
   LZ4 with executable code).   
      
   Main reason binaries are LZ compressed is mostly to make them load   
   faster from an SDcard. Similarly, no Deflate both for complexity reasons   
   and because Deflate would be slower to decompress than the speed of the   
   SDcard.   
      
      
      
   Address Space:   
      Monolithic 48 (flat) or 96 bit (segmented / "quadrants");   
        In the 96 bit mode, only the low 48b is linear;   
        Another 48 bits functions as the quadrant address.   
      
      
   In 96-bit mode, each program can be given its own local 48-bit quadrant   
   while still being part of a larger shared 96 bit space, and it is   
   possible to use 128-bit pointers to address anywhere inside this space.   
      
   However, the default pointer size remains as 64-bits with 48-bits for   
   the address and 16-bit for tag metadata (usually zeroed for bare   
   pointers in C, or non-zero for bounds-checked pointers; also usable for   
   dynamic type-tag handling).   
      
   When using 64-bit pointers, it is functionally similar to memory   
   addressing in the 65C816 (or segmented addressing in x86, though this is   
   a less accurate comparison). Mostly this is done because using 128-bit   
   pointers as the default would be overkill (in C, the pointers use a   
   "__huge" modifier, Say: "void * __huge ptr;").   
      
      
   So, with 64-bit pointers, one can access up to 256TB, or 64 RB/RiB   
   (ronnabyte/robibyte, with the newer SI prefixes), or 256 TQu   
   ("teraquad", if one assumes each "quad" is 256TB).   
      
      
      
   In 48-bit mode, memory management is via a 3-level page-table (with a   
   16K page-size and 64-bit page-table entries).   
      
   In 96-bit mode, the page-tables are mostly replaced by B-Trees or   
   "hybrid trees" where the high-order levels are managed by a B-Tree, but   
   the last 1 or 2 levels are page-tables.   
      
   Partly this is because 3-level page tables are reasonably efficient for   
   a 48-bit space, but quickly become very inefficient with a large   
   sparsely filled address space (with an 8-level page-table, most upper   
   level pages are almost entirely empty).   
      
   Also, by various metrics, the B-Tree won out against its nearest   
   competitors, namely an AVL Tree and a Hash-Table Tree. A hash-table can   
   potentially be faster in simple cases, O(1) best-case vs O(log2 n), in   
   experimental tests the B-Tree retained better "average case" performance   
   (as the node got closer to full hash-table speed falls in the toilet).   
   In terms of both speed and memory use, the B-Tree beat out the AVL Tree.   
      
      
   Memory Protection:   
   Still not well developed in practice, but in theory, there is ACL   
   checking on memory pages (so, for example, a task PID or similar be can   
   be checked against an ACL to determine whether it has been granted a   
   particular access to a particular ACLID), with the CPU throwing an   
   access-fault if the program tries to access memory that it has not been   
   granted access. Each memory page may be assigned to an ACL.   
      
   This partly extends an older system known as VUGID which applied   
   (non-ACL) User/Group/Other checks to pages (using a similar model to   
   that used in the Unix/Linux filesystems). Conceptually, an ACL is more   
   general (and has some similarities to file protection in Windows / NTFS).   
      
      
   A lot of this is partially built into the MMU; Though, in my case I am   
   using a software-managed TLB and ACLB (some of this would likely be   
   impractical if all of the memory management were done entirely in hardware).   
      
      
   Note that despite some overlap (and both systems having support for   
   bounds-checking pointers), the abstract memory model differs   
   significantly from something like CHERI. However, I personally suspect   
   that my design is "more practical" as it doesn't require using larger   
   pointers or mess up traditional C coding practices (the constraints   
   imposed by CHERI are, within the limits of the letter of the C standard;   
   a notable departure from the "common expectations" of how C code works   
   on typical flat-addressed machines).   
      
   While one could argue similar about the 96b address space, likely 99.9%   
   of programs can probably ignore that anything exists outside the low 48   
   bits (it's local address space just floating at some random location   
   within an endless black sea).   
      
      
      
   Filesystem:   
   Mostly FAT32 for now, mostly as Windows interferes with me using   
   anything. Had experimented with hacking a few Unix-like features onto   
   FAT using a vaguely similar approach to UMSDOS albeit still using LFNs   
   (the internal-use SFN is keyed to any additional file metadata though).   
      
      
   HAL API:   
   I have an API known as TKGDI which provides some interfaces for hardware   
   interfacing. It sort of resembles a hybrid of Windows GDI, MMSYS, and   
   VFW. It can also export an OpenGL interface.   
      
   It provides a C style API, but internally works using a mechanism   
      
   [continued in next message]   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)