I</th>	T</th>	E</th>	O</th></tr></thead>
0</td>	0</td>	0</td>	0</td></tr>
0</td>	0</td>	1</td>	1</td></tr>
0</td>	1</td>	0</td>	0</td></tr>
0</td>	1</td>	1</td>	1</td></tr>
1</td>	0</td>	0</td>	0</td></tr>
1</td>	0</td>	1</td>	0</td></tr>
1</td>	1</td>	0</td>	1</td></tr>
1</td>	1</td>	1</td>	1</td></tr> </tbody></table> </li> </ul> </li> </ol> While the above example doesn't really use an SMT solver's true power, it does</em> illustrate the fundamentals of how to interface with one. We will discuss how to use other datatypes and theories with SMT-LIB as we turn a logic puzzle into an SMT query.</p> SMT And Logic Puzzles</h2> With the whirlwind introduction to SMT out of the way, my goal with this post is to show you how to map the word problem presented by a logic puzzle to a `SAT</code>isfiable SMT query. I will introduce an example logic puzzle first, and then show how to use various SMT-LIB commands to create some datatypes and constraints to represent the puzzle.</p>` `At a high level, if you visualize a logic puzzle grid, such as the one below, the conversion process looks like this:</p>` The SMT theory of algebraic datatypes will be used to represent the headers of each row and column of data.</li> Correlating each row to the correct columns of data will be represented by the theory of uninterpreted functions .</li> Clues/hints and constraints implicit</em> in a logic puzzle will be represented by assert</code>s.</li> </ul> This empty logic puzzle contains 9 rows and columns, along with headers for each row and column. The column headers, in left-to-right order, are: Batman, Jake, Dibii, Lazer, Sleep, Ball, 1, 2, and 3. The row headers in top-to-bottom order, are: Ruby, Spot, and Starbuck, 1, 2, 3, Lazer, Sleep, and Ball.</p> Each set of 3 row and column headers belong to the same category</em>:</p> Batman, Jake, Dibii are tom</em>cats.</li> Ruby, Spot, and Starbuck are queen</em></a> cats.</li> Lazer, sleep, and ball are activities</em>.</li> 1, 2, and 3 are number of kittens</em>.</li> </ul> To solve the puzzle, you are intended to mark each table entry</em> a ✓ if the row and column are associated with each other, or ✗ if the row and column are not</em> associated with each other, based on clues given to you (not represented in the above image).</p> A table entry whose row and column are of the same category does not need to be marked. The pictured table represents this by completely omitting such entries, making the table similar to an upper triangular matrix</a>.</p> The bottom part of the puzzle consists of a table with the columns queen</em>, tom</em>, activity</em>, and number of kittens</em>. The column queen</em> is filled in for you with Ruby, Spot, and Starbuck. You fill the remaining rows of this table based on which row and columns you marked with a ✓ in the above table.</p> I got the logic puzzle template (before filling in the headers) from Daydream Puzzles</a>.</figcaption> </figure> An Example Puzzle</h2> For this post, I'll be using a puzzle from Aha! Puzzles</a> as an example. I'm not going to copy the puzzle description, since a person presumably spent time to create the puzzle (among others), and I don't want to steal their work. Instead I will paraphrase the puzzle and grid, which describes a set of cats.</p> Description</h3> There are three queens</a> named Ruby</em>, Spot</em>, and Starbuck</em>.</li> Each queen has a partner, favorite activity, and number of kittens. As is customary for this type of problem, it is implied that each Queen has a different partner, favorite activity, and number of kittens.</strong></li> The possible partners (toms, as in tom</em>cats) are: Batman</em>, Jake</em>, and Dibii</em>.</li> Activities include: chasing a Lazer</em>, Sleep</em>ing, and chasing a Ball</em>.</li> Each queen/tom pair has between 1</em> to 3</em> kittens (inclusive).</li> </ul> Given the above information, and the following clues, figure out each queen's partner, favorite activity, and the number of kittens they have.</p> Clues</h3> The puzzle is intended to have a unique answer</em>. However, using only the description above, there are multiple correct answers. The clues below are intended to limit the multiple correct answers down to a single unique answer^{6</sup></a> :</p>} Batman's partner likes to chase a ball. She was not Starbuck.</li> Dibii's partner liked the lazer.</li> Ruby was a cat who liked to sleep.</li> Starbuck has two more kittens than whoever Batman's companion is.</li> </ol> Mapping The Puzzle To An SMT Query</h2> Now that we have a puzzle, we can map the textual description to a query and a mathematical model that the SMT solver will return. Let's create the query piece by piece, starting with initial setup code:</p> (set-option :produce-models true) (set-logic ALL) </code></pre> The first two lines we've already seen, but we have a new logic ALL</code>. set-logic ALL</code> is a bit of a cop-out; SMT-LIB allows ALL</code> for applications which generate queries on-the-fly. ALL</code> is not intended for handwritten queries. Unfortunately, z3</code> doesn't</a> really</a> have that many logics that support datatypes. And none of the logics that do</em> support datatypes seem to support both</em> integers and uninterpreted functions.^{7</sup></a> So, without any better options for now, I'm just making explicit that z3</code> should use "whatever means necessary" to solve the logic puzzle}^{8</sup></a> .</p>} Enums</h3> declare-datatypes</code> is part of the most recent version of SMT-LIB, version 2.6. It is a very general function that allows you to create complex data types, including recursive</a> data types. In our usage we will only be using it to create the equivalent of enum</code></a>s without data in other programming languages^{9</sup></a> .</p>} In general, you create an enum</code> Foo</code> whose variants</em> A</code>, B</code>, C</code> (and more, if wish) have data using the following syntax^{10</sup></a> :</p>} (declare-datatypes ((Foo 0)) (((A) (B) (C)))) </code></pre> Anywhere in your query where a value of type Foo</code> is used, the SMT solver has no choice but to make the value either A</code>, B</code>, or C</code>. Additionally, the following properties hold^{11</sup></a> :</p>} A = A</code>, B = B</code>, and C = C</code> (reflexivity</a>)</li> A != B</code>, B != C</code>, and A != C</code> (antisymmetry</a>).</li> </ul> This can be confirmed with the following query, which returns SAT</code>:</p> (declare-datatypes ((Foo 0)) (((A) (B) (C)))) (assert (and (= A A) (= B B) (= C C))) (assert (and (distinct A B) (distinct A C) (distinct B C))) (check-sat) </code></pre> With the above syntax and properties in mind, the names of the cats and their activities correspond very nicely to enum</code>s whose variants don't have data:</p> (declare-datatypes ((Tom 0)) (((Batman) (Jake) (Dibii)))) (declare-datatypes ((Queen 0)) (((Ruby) (Spot) (Starbuck)))) (declare-datatypes ((Activities 0)) (((Lazer) (Sleep) (Ball)))) </code></pre> Uninterpreted Functions</h3> Functions form the backbone of programming languages and mathematics, and SMT formulas are no exception. We can create pure functions</a> to abstract away common parts of our query using define-fun</code>. Here is an example add</code> function that "adds one to its input Int</code>":</p> (define-fun add ((x Int)) Int (+ x 1)) (check-sat) (get-value ((add 1) (add 2) (add 10))) </code></pre> The above query returns SAT</code></a> in response to check-sat</code>. After check-sat</code>, your model has an add</code> function that the solver can call.</p> We don't have any assert</code>s in the above query, so get-model</code> returns nothing. However, we can display the result of calls to our add</code> function using get-value</code>. get-value</code> takes a single argument, but this single argument can take an arbitrary number of terms</em>. Each of the three terms to the get-value</code> call above will make the solver read, evaluate, and print</a> the results of calling your add</code> function on various integers- exciting!</p> sat (((add 1) 2) ((add 2) 3) ((add 10) 11)) </code></pre> Not only can we define functions, we can also declare</em> functions, without bothering to define/fill in the function body! We can request the solver to fill in the function body for us by using declare-fun</code>:</p> (declare-fun add_two (Int) Int) (check-sat) (get-model) (get-value ((add_two 1))) </code></pre> The above query returns sat</code> in response to check-sat</code>. get-model</code> will then return the definition</em> of add_two</code> that the solver created:</p> sat ( (define-fun add_two ((x!0 Int)) Int 0) ) (((add_two 1) 0)) </code></pre> In this case add_two</code> returns the literal 0</code>. Not particularly interesting. What if we add an assert</code>, to force add_two</code> to return 3</code> when given the input 1</code>?</p> (declare-fun add_two (Int) Int) (assert (= (add_two 1) 3)) (check-sat) (get-model) (get-value ((add_two 1))) </code></pre> sat ( (define-fun add_two ((x!0 Int)) Int 3) ) (((add_two 1) 3)) </code></pre> It's a function that returns 3</code> for all inputs. Now the solver is being cute. "Technically correct" is the best type of correct. The solver won't create a more interesting function body unless forced to via proper constraints on the input arguments and return values^{12</sup></a> .</p>} We will constrain our uninterpreted functions shortly, but for now, let's declare 4 uninterpreted functions with declare-fun</code>. These functions will in fact return the answers for our logic puzzle, once the solver fills them in:</p> (declare-fun QueenPartner (Queen) Tom) (declare-fun QueenActivity (Queen) Activities) (declare-fun QueenKittens (Queen) Int) (declare-fun TomPartner (Tom) Queen) </code></pre> To be specific:</p> QueenPartner</code> returns the Tom</code> partner of the given Queen</code>.</li> QueenActivity</code> returns the given queen's favorite activity.</li> QueenKittens</code> returns the number of kittens a given Queen</code> has as an Int</code>.</li> TomPartner</code> returns the Queen</code> partner of the given Tom</code>.</li> </ul> Implicit Constraints</h3> By their very nature, logic puzzles contain some usually-implied rules</em> that we must follow when solving them if we don't want incoherent, nonsensical answers:</p> Each possible choice in each category is different</em> from all the other choices. It should be apparent that, for instance, Batman and Dibii are different tomcats (uniqueness</em>).</li> There are a finite number of choices for each category, and this number is the same for each category. E.g. each of the three</em> queens can choose from one of three</em> activities. There is no fourth "secret" activity that the logic puzzle knows about and we don't (finiteness</em>).</li> As mentioned earlier</a>, each queen has a mutually-exclusive partner, number of kittens, and activity with respect to other queens.</li> We can also deduce that a tom has the same number of kittens and activity as its queen partner.</li> </ul> A logic puzzle grid such as above</a> visually makes it easy to follow these rules. Unfortunately, an SMT solver knows nothing about logic puzzles without our help, so we must explain those implicit constraints</em> in our query. We use assert</code>s to force an SMT solver to reject nonsensical-but-valid answers to a query of variables, where in the case of a logic puzzle, our variables are our uninterpreted functions.</p> Our enum</code>s we created via declare-datatype</code> already encode these uniqueness/finiteness constraints mentioned above. We can also represent the number of kittens each cat pair has as a NumKitten</code>s enum</code>, but for convenience that will become clear shortly, I chose to use an Int</code>. While Int</code>s have a uniqueness constraint (1</code> is different than 2</code>), in SMT-LIB, Int</code>s do not have a finiteness constraint. Thus without additional information, the solver free to set NumKitten</code>s for each cat pair to 1</code>, 10</code>, or even 1,234,567</code>, among other fine Int</code>s, if the solver so chooses.</p> The puzzle's description implies</del> outright says that the number of kittens each cat pair must be between 1</code> and 3</code> inclusive. We can constrain the values for number of kittens the solver will try using an assert</code>:</p> ; Integer constraint (assert (and (>= (QueenKittens Ruby) 1) (<= (QueenKittens Ruby) 3) (>= (QueenKittens Spot) 1) (<= (QueenKittens Spot) 3) (>= (QueenKittens Starbuck) 1) (<= (QueenKittens Starbuck) 3))) </code></pre> While the uniquess constraints of enum</code>s and Int</code>s gets us far, it's not quite enough. As quoted above</a>:</p> As is customary for this type of problem, it is implied that each queen has a different partner, favorite activity, and number of kittens.</strong></p> </blockquote> Although the solver recognizes that each tom is a different cat, without additional information, nothing prevents the solver from giving an answer which assigns, for instance, two queens to the same tomcat partner.</p> We can further constrain the solver from trying to assign two queens the same partner, number of kittens, and activity by constraining the uninterpreted functions</a> we defined earlier via more assert</code>s. For instance, the statement (assert (not (= (QueenPartner Ruby) (QueenPartner Spot)))</code> tells the SMT solver "However</em> you decide to fill the function body, I need the function QueenPartner</code> to not</em> return the same value for Ruby</code> and Spot</code>". To properly constrain our query, you need one assert</code> for each possible pair of values in each category, which can be combined into a single large conjuction</a>^{13</sup></a> :</p>} ; "Only one dot in each row/col" consistency check. (assert (and (not (= (QueenPartner Ruby) (QueenPartner Spot))) (not (= (QueenPartner Ruby) (QueenPartner Starbuck))) (not (= (QueenPartner Spot) (QueenPartner Starbuck))) (not (= (QueenKittens Ruby) (QueenKittens Spot))) (not (= (QueenKittens Ruby) (QueenKittens Starbuck))) (not (= (QueenKittens Spot) (QueenKittens Starbuck))) (not (= (QueenActivity Ruby) (QueenActivity Spot))) (not (= (QueenActivity Ruby) (QueenActivity Starbuck))) (not (= (QueenActivity Spot) (QueenActivity Starbuck))))) </code></pre> The above check is morally equivalent to a human making sure a logic grid has only one ✓ per row and column for each combination</a> of values in each category. Because of the uniqueness constraints for each enum</code> and Int</code>, the above is sufficient to make sure the solver knows that each queen has a mutually exclusive partner, number of kittens, and activity and that such a solution exists</em>^{14</sup></a> .</p>} Lastly, each cat pair shares a favorite activity and the number kittens they have. So far, we've only written constraints for each queen in the cat pair. However, we don't need to rewrite the constraints for each tom! There is another consistency check we need, and a side-effect, we can represent each tom's favorite activity and number of kittens for free in terms of this constraint.</p> The TomPartner</code> uninterpreted function returns the Queen</code> who is the partner of the given Tom</code>. Likewise, the QueenPartner</code> function returns the Tom</code> whose partner is the given Queen</code>. Without extra constraints, the SMT solver doesn't know "obvious" facts like "'the partner of the partner' of a cat is the same cat you started with"^{15</sup></a> . We can represent this fact with another assert</code>:</p>} ; All other constraints for Tom are implied if the partner of the partner is the same. (assert (and (= (QueenPartner (TomPartner Batman)) Batman) (= (QueenPartner (TomPartner Jake)) Jake) (= (QueenPartner (TomPartner Dibii)) Dibii))) </code></pre> After this constraint, TomPartner</code> unambiguously maps</a> a tom back to queen. Since a cat pair has the same favorite activity and number of kittens, the solver now has enough information to solve for each tom's favorite activity and number of kittens in terms of queens.</p> Explicit Constraints</h3> The implicit constraits are necessary but not sufficient</a> to constrain the logic puzzle to a unique answer. If an SMT solver is presented with only the above lines, but there will be multiple ways to fill in our uninterpreted functions which satisfy that query. The solver will choose to present to you^{16</sup></a> one interpretation for each function out of all valid ones, and chances are, it won't be the intended answer to the logic puzzle based on the clues.</p>} If you incorporate the clues, however, the SMT solver should</em>^{17</sup></a> be limited to only one valid answer to your query. In this sense, the clues given}above</a> form the explicit</em>, outright-stated constraints unique to a specific logic puzzle. Explicit constraints are handled identically to the implicit constraints in the previous section</a>, by mapping a textual description of the clues described earlier to asserts</code>:</p> Batman's partner likes to chase a ball. She was not Starbuck.</p> </blockquote> ; 1. Batman chose the female who liked to chase a ball, but she was not Starbuck. (assert (= (QueenActivity (TomPartner Batman)) Ball)) (assert (not (= (TomPartner Batman) Starbuck))) </code></pre> Dibii's partner liked the lazer.</p> </blockquote> ; 2. Dibii's companion liked to chase the laser light. (assert (= (QueenActivity (TomPartner Dibii)) Lazer)) </code></pre> Ruby was a cat who liked to sleep.</p> </blockquote> ; 3. Ruby loved to cuddle up to her male for a long afternoon nap in the sun. (assert (= (QueenActivity Ruby) Sleep)) </code></pre> Starbuck has two more kittens than whoever Batman's companion is.</p> </blockquote> This clue is the reason I represented the number of kittens as an Int</code>. While I would have to reinvent addition if I used a NumKittens</code> enum</code>, the SMT solver already knows the theory of Int</code>s and can add them using +</code>^{18</sup></a> .</p>} ; 4. Starbuck had two more kittens than Batman's companion. (assert (= (QueenKittens Starbuck) (+ (QueenKittens (TomPartner Batman)) 2))) </code></pre> Getting An Answer</h3> Once, we add check-sat</code>, our query text file is finished, and we can feed it to z3</code> using the invocation described</a> to check whether our query is satisfiable. You can download the completed query here</a> and run it online</a> (as of 2023-08-13) if you wish.</p> Many times, knowing SAT</code> or UNSAT</code> is "good enough". But for a logic puzzle, we actually want the variable values. get-model</code> works, but this shows the definition of the uninterpreted functions you asked the solver to fill in. While seeing the function bodies are useful information, for solving a logic puzzle, it's probably better to use get-value</code> to see individual outputs of each now</em>-interpreted function for "interesting" inputs.</p> To compare and contrast, I use all of check-sat</code>, get-model</code>, and get-value</code>:</p> (check-sat) (get-model) (echo " Spot:") (get-value ((QueenPartner Spot) (QueenKittens Spot) (QueenActivity Spot))) (echo " Ruby:") (get-value ((QueenPartner Ruby) (QueenKittens Ruby) (QueenActivity Ruby))) (echo " Starbuck:") (get-value ((QueenPartner Starbuck) (QueenKittens Starbuck) (QueenActivity Starbuck))) </code></pre> Running the entire query through my local copy of z3</code> gives the following output:</p> sat ( (define-fun QueenKittens ((x!0 Queen)) Int (ite (= x!0 Spot) 1 (ite (= x!0 Starbuck) 3 2))) (define-fun QueenPartner ((x!0 Queen)) Tom (ite (= x!0 Spot) Batman (ite (= x!0 Starbuck) Dibii Jake))) (define-fun QueenActivity ((x!0 Queen)) Activities (ite (= x!0 Starbuck) Lazer (ite (= x!0 Ruby) Sleep Ball))) (define-fun TomPartner ((x!0 Tom)) Queen (ite (= x!0 Jake) Ruby (ite (= x!0 Dibii) Starbuck Spot))) ) Spot: (((QueenPartner Spot) Batman) ((QueenKittens Spot) 1) ((QueenActivity Spot) Ball)) Ruby: (((QueenPartner Ruby) Jake) ((QueenKittens Ruby) 2) ((QueenActivity Ruby) Sleep)) Starbuck: (((QueenPartner Starbuck) Dibii) ((QueenKittens Starbuck) 3) ((QueenActivity Starbuck) Lazer)) </code></pre> First off, the query is sat</code>, which is a good start! The model returned by get-model</code> isn't hard for me to follow either. I could fairly quickly figure out each answer to the logic puzzle using get-model</code> alone, and I'm sure anyone else reading this post could too. But using get-value</code> to evaluate the functions the solver filled in for us gives us the logic puzzle answers directly, ready to be written down! So let's save ourselves the trouble and populate the puzzle's answers thanks to get-value</code>!</p> Was the solver correct? Well, as of this writing (2023-07-23), Aha! Puzzles lets you solve logic puzzles interactively online. I'll just let the below screenshot speak for itself:</p> The popup says "Congratulations, you win!", so indeed z3</code> was correct. Although the logic puzzle grid and answer table is obscured by the popup, there is just</em> enough of the grid and answer table shown to verify that I filled in the puzzle with the answers returned by z3</code>.</figcaption> </figure> Conclusion/Next Steps</h2> I still find logic puzzles fun to work through manually. To me, asking a machine to solve every logic puzzle for you defeats the satisfaction of working your brain to solve the puzzle yourself. So while I hope the above example provides some insight as to how to map word problems to an SMT query/model, I wouldn't suggest that you start solving every</em> logic puzzle with an SMT solver.</p> That said, I found that trying to map a logic puzzle to an SMT solver query was also</em> intellectually stimulating. It is a low-stakes problem that gave me the opportunity to exercise my skills at creating SMT queries, as opposed to my typical usage where I have a program generate</a> queries and models for me. I was excited after I got my query working, and I wanted to document the result.</p> Speaking of generating queries and models, it's possible to take this logic puzzle-solving use case further by creating a Domain-Specific Language</a> (DSL). Like how the previously-linked SymbiYosys converts Verilog to SMT-LIB queries, this DSL would convert a more convenient textual representation of a logic puzzle into SMT-LIB queries. It's a bit more up-front work than writing SMT-LIB queries manually, and probably not worth the time spent other than for learning. However, since this project is meant</em> to be low-stakes and for fun, I may look into a DSL in the future for a follow-up post. It's about time I wrote a(nother</a>) compiler anyway.</p> Thanks</h2> I would like to thank my friends Screwtape</a> and Alyssa Rosenzweig</a> for their general feedback and a few last-minute corrections on the final draft of this post.</p> Footnotes</h2> 1</a> I will not be discussing how to write a SAT solver; for those interested I suggest this tutorial</a> for building a SAT solver in Python. The tutorial is based on Donald Knuth's</a> "SAT0W" solver.</p> Although I have not personally done so yet, I would probably also look at Knuth's "SAT10" solver, which is based on the Davis–Putnam–Logemann–Loveland</a> (DPLL) algorithm.</p> 2</a> To be honest, I'm going by what Wikipedia says</a> here. There's no external reference: Early attempts for solving SMT instances involved translating them to Boolean SAT instances (e.g., a 32-bit integer variable would be encoded by 32 single-bit variables with appropriate weights and word-level operations such as 'plus' would be replaced by lower-level logic operations on the bits) and passing this formula to a Boolean SAT solver. </blockquote></p> 3</a> Just like for SAT solvers, I will not be discussing how to write an SMT solver. While in the process of writing this post, I have been learning about the DPLL(T) framework myself. I've found the following set of slides useful:</p> Lecture 7</a> of Emina Torlak's</a> CSE507</a>. In particular, Emina's slides outline the DPLL(T) algorithm in terms of Conflict-Driven Clause Learning</a> (CDCL), an extension of DPLL.</li> Lecture 2</a> of Wes Weimer's</a> CS590</a>. </ul> </p> 4</a> In my own experience, yices</code> is significantly faster at many problems involving formal verification of HDL. However, it is not</em> universal. This is why the Symbiyosys</a> front end for HDL verification will run the same query against multiple SMT solvers in parallel if you have them installed.</p> 5</a> If you don't include parentheses to actually evaluate operators, weird things you probably didn't intend will happen. For instance, try comparing the result of the query (assert (= or true false true))</code> to (assert (= (or true false) true))</code>. </p> 6</a> More succinctly, the clues "constrain the solution space".</p> 7</a> The logic that's closest to what I need, Quantifier-Free Finite Domain</a> (QF_FD</code>), has some additional restrictions</a> which cause my logic puzzle query to return UNKNOWN</code>. I'm not actually certain why it returns UNKNOWN</code>, but I suspect</em> I'm running afoul of the restrictions by using datatypes with uninterpreted functions, and not just</em> equality (=</code>) comparisons.</p> 8</a> I belive `ALL` is z3</code>'s default logic anyway.</p> 9</a> declare-datatypes</code> is not required, and the logic puzzle can be solved without them. In fact, I have</em> constructed a working query without datatypes</a> for this logic puzzle- twice</a>! I wanted to solve the puzzle with yices</code> as well as z3 as an exercise; while yices</code> does support enum</code>s and the like, they are not compatible with SMT-LIB.</p> As shown in the linked yices</code> examples, declare-sort</code>, declare-const</code>, and declare-fun</code>, plus some assert</code> constraints can emulate declare-datatypes</code> and finite integers reasonably. However, for a handwritten example, I wanted to focus on creating constraints appropriate to a logic puzzle rather than reinventing enum</code>s.</p> </p> 10</a> The integer argument to declare-datatypes</code> is used for parametric polymorphism</a>. It represents, informally, "the number of types to be filled in later".</p> 11</a> What A</code>, B</code>, and C</code> actually are</em> in terms of "bits in memory somewhere" doesn't matter as long as the solver manipulates these values of type Foo</code> according to the above rules.</p> 12</a> To make the example more interesting, I tried to use assert</code>s to get the solver to generate a function body which uses +</code> (integer addition), but got bored and gave up. This post is getting long anyway.</p> 13</a> Readers familiar with using z3</code> may notice that I can more-succinctly represent the exclusivity constraints using a forall</code> quantifier</a>. Indeed, this works, and was how I originally</a> solved the example logic puzzle. Except for an extra pair</code> datatype</a>, the quantifier version of our query is very similar to the quantifier-free query</a> constructed in this blog post; just compare the comments. However, I decided not</em> not to discuss quantifiers in this post for a few reasons: The quantifiers are not required for enumerations with finite variants to create the constraints.</li> I am not exceptionally familiar with using quantifiers in SMT queries and when they are actually required</em>. I haven't needed them for HDL verification, for instance.</li> Quantifiers complicate</a> the SMT solver's job. While queries without quantifiers can always be determined to be SAT</code> or UNSAT</code>, some queries with quanitifers are undecidable</a>. As of this writing (2023-07-02) I don't think I'm qualified to discuss undecidability other than in passing. </li> Not all SMT solvers support quantifiers (e.g. yices</code>).</li> </ol> Therefore, while the quantifiers are likely harmless here, I'd rather them when they are actually required</em>, not just as a means to make queries take fewer lines.</p></p> 14</a> If two enum</code> variants could be equal each other in value (i.e. not distinct</code>), it would be impossible for our uninterpreted functions to return three distinct</code> enum</code> variants to three different values of input enum</code> variants. To show this, play with this commented query online</a>: (declare-sort Foo 0) ; declare-sort declares a new type, like `Int`, `Bool`. ; It has fewer assumptions baked-in than `declare-datatypes`. ; Create some named values for our types. ; These are like our `enum` variants. (declare-const foo_one Foo) (declare-const foo_two Foo) (declare-const foo_three Foo) (declare-sort Bar 0) (declare-const bar_one Bar) (declare-const bar_two Bar) (declare-const bar_three Bar) ; Our uninterpreted function. Takes a `Foo`, returns a `Bar`. (declare-fun my-fun (Foo) Bar) ; Ensure each constant has a unique value. This is implied ; `declare-datatypes`. ; ; Change _any_ of the 6 asserts from `distinct` to `=`. ; The solver will return `unsat`. (assert (distinct foo_one foo_two)) (assert (distinct foo_one foo_three)) (assert (distinct foo_two foo_three)) (assert (distinct bar_one bar_two)) (assert (distinct bar_one bar_three)) (assert (distinct bar_two bar_three)) ; Prevent the solver from trying to create more values for our types ; from thin air. This is implied in `declare-datatypes`. (assert (and (or (= (my-fun foo_one) bar_one) (= (my-fun foo_one) bar_two) (= (my-fun foo_one) bar_three)) (or (= (my-fun foo_two) bar_one) (= (my-fun foo_two) bar_two) (= (my-fun foo_two) bar_three)) (or (= (my-fun foo_three) bar_one) (= (my-fun foo_three) bar_two) (= (my-fun foo_three) bar_three)))) ; Each input to my-fun should return a unique output. ; This is impossible unless foo_{one, two, three} have ; different values according to `=`. (assert (distinct (my-fun foo_one) (my-fun foo_two))) (assert (distinct (my-fun foo_one) (my-fun foo_three))) (assert (distinct (my-fun foo_two) (my-fun foo_three))) (check-sat) (get-model) </pre></p> 15</a> While I praise the SMT solver for being open-minded, polyamory is not within the scope of this logic puzzle.</p> 16</a> By default an SMT solver will give you one</em> answer. It is possible to get all</em> possible answers to an SMT query. Since a logic puzzle is intended to have a single answer, ensuring that the number of unique answers is 1 would be another good consistency check at the cost of execution time. Unfortunately, I don't remember how to do this.</p> 17</a> The puzzle should</em> have a unique answer if it was designed properly.</p> 18</a> I chose to write the assert</code> in terms of Int</code> and +</code> as an example. When numbers you can choose from in a category aren't sequential, and/or the clues are more complicated, I think it's useful to be able to model clues with other theories the SMT solver provides.</p> In this case, because we know that NumKittens</code> is constrained from 1</code> to 3</code> inclusive, you could just as easily derive the constraint as: (assert (= (QueenKittens Starbuck) 3)) (assert (= (QueenKittens (TomPartner Batman)) 1)) </pre></p> A Tour Of ethflop 2019-12-11T00:00:00+00:00 A Tour Of ethflop</h1> ethflop</a> is a new (2019!) Terminate and Stay Resident</a> (TSR) program for IBM PCs and compatibles running a DOS flavor. ethflop will convert either your drive A:\ or B:\ floppy drive into a networked drive that uses Ethernet frames to talk to a Linux server. The Linux server runs a daemon called ethflopd</code> that can supply floppy images in place of physical disks.</p> I first found out about ethflop from a retweet</a> of the FreeDOS Project's account. I knew then and there I at least</em> had to try this out. It's like I was the target audience! And if you:</p> Have old hardware "optimized" for running DOS.</li> Enjoy (responsibly or otherwise!) networking your old computers to other old and new computers.</li> Have a large number of floppy images that you downloaded over the years.</li> </ul> then I think ethflop is worth looking at! I imagine a number of vintage-oriented people who would be happy with having access to a bunch of floppy images from their DOS machine without needing to write them to physical floppies!</em> So I've decided to take a look myself and write down my initial experiences using ethflop.</p> My Hardware Setup (For Testing)</h2> ethflop requires two machines to use: a Linux server which runs a daemon (ethflopd</code>) that has access to some floppy images, and a machine running DOS which interfaces to the Linux daemon to access said floppy images (using the TSR portion of ETHFLOP.COM</code>). I present the hardware setup I used in the next two sections for showing off my PC AT</del> giving the rest of this post context</ins>. Feel free to skip</a> this section and refer back to it when necessary; it is mostly inessential.</p> My 286 (DOS) Setup</h3> My 286 is an early IBM PC AT running at 6 MHz. It has 2 floppy drives- drive A:\ is a 1.2MB High Density drive, and drive B:\ is a 360kB Double Density drive. Additionally, my AT has one full height hard drive at drive C:\ (the original CMI</a> hard drive), and one half-height hard drive at D:\ that sits rather snugly in the slot under the 2 floppy drives. The C:\ drive is the boot drive and the D:\ drive is used mainly for receiving temporary files that I subsequently forget to clean up :).</p> Picture of my IBM PC AT, showing both floppy drives. The screen includes some DOS commands I ran which are relevant to this blog post, including unloading the DOSED</code></a> autocompleter (as it interferes with CTTY</code>), loading a packet driver, and then changing the console to a serial port via CTTY</code>.</figcaption> </figure> Because of the way ethflop works, you lose access to one of the floppy drives; I chose to use drive B:\ with ethflop for testing purposes. The hard drives are unaffected, and I've yet to test whether you can reliably use ethflop with a boot floppy. If you do not have a physical drive B:\, you must use drive A:\ with ethflop.</em> This is because DOS will emulate a virtual B:\ drive using drive A:\ if no physical drive is detected, and no B:\ drive requests will ever be sent to ethflop.</p> My Network Interface Card (NIC) is a 3Com 3c509, which is still a fairly common ISA</a> NIC that can be even be found for sale off Ebay. 3Com still had packet drivers on their website for the 3c509 when I was searching for them in late 2010, but they pulled an Intel and they are long gone. Fortunately, the Internet Archive comes through</a> yet again!</p> Due to me misunderstanding the Ebay listing at the time, I bought a model of 3c509 which only has thinnet (10BASE2)</a> and thicknet (10BASE5)</a> connections. Fortunately, 10BASE2 to 10BASE-T converters are still common- if way too expensive- and occassionally one can buy an old hub off Ebay with the same hardware^{1</sup></a> as these converters. I'm not in a position to really offer concrete advice on setting up 10BASE2 right now. The moral here is get an ISA NIC with an RJ-45 connector, which is the}twisted pair (10BASE-T)</a> Ethernet that you know and love.</p> Linux Setup</h3> I used my headless Tinker Board</a> to run the Linux daemon. My distro is DietPi, though I imagine the distro really doesn't matter; the source to ethflopd</code> is contained in a single C file and I had no troubles compiling and running it. Notably, I'm only using the Tinker Board's wifi, and I can confirm ethflopd</code> will work if used with a wlan</code> interface.</em> I'm assuming my router takes care of converting from raw Ethernet frames to 802.11 frames, because they are not compatible</a>.</p> This image shows a screen capture of an SSH session with the Tinker Board. The listing shows I am in the floppy image directory and I have a few floppy images prepared to share with the PC AT. I couldn't think of anything better to show off a running headless system :).</figcaption> </figure> For ease of testing, I am interfacing to DOS using a serial port and minicom</code> on the Tinker Board. It is possible to redirect DOS input and output to a serial port (COM2</code> in my case) using the CTTY</code> command. Using minicom</code> makes it much</em> easier to capture DOS command output compared to taking photos. The picture of my AT above was taken just</em> after I invoked CTTY</code>.</p> Compiling and Installing</h2> Before using ethflop, one has to compile and install it first. I didn't have any issues with compiling and installing, but I've documented my experiences on how to do it for others just in case.</p> Compiling and Installing ethflopd</code></h3> To start, download the source distribution from the ethflop website</a> using wget</code> or other means. I recommend downloading the source to a Linux machine. Once you unzip (unzip -d ethflop ethflop-20191003.zip</code>) the source distribution, you will see a directory like this:</p> wjones@DietPi:~/src/ethflop$ ls -l total 120 -rw-r--r-- 1 wjones wjones 41865 Oct 3 16:48 ethflop.asm -rw-r--r-- 1 wjones wjones 3696 Oct 3 17:04 ethflop.com -rwxr-xr-x 1 wjones wjones 18568 Nov 25 22:15 ethflopd -rw-r--r-- 1 wjones wjones 38305 Oct 3 16:50 ethflopd.c -rw-r--r-- 1 wjones wjones 6192 Oct 3 16:55 ethflop.txt -rw-r--r-- 1 wjones wjones 803 Oct 3 17:02 Makefile </code></pre> The ETHFLOP.COM</code> tsr is written in nasm</code></a>-style assembly. While nasm</code> is provided by many Linux package repos, the author provided an already-assembled .COM</code> file for convenience. You will need to transfer this file to your DOS machine.</p> The ethflopd</code> portion of ethflop needs to be compiled manually. Type make</code> and you will get a binary at ethflopd</code>. Any recent Linux should be able to run ethflopd</code>. As of this writing (12-8-2019), I have not tested the daemon on non-Linux systems.</p> There's no install target, so you need to copy ethflopd</code> to its final location manually. In my case, I copied the file to my private bin directory: cp ethflopd ~/.local/bin</code>.</p> Lastly, create a storage directory for your floppy images: mkdir ~/src/img</code>. This is the directory ethflopd</code> will use to present virtual floppy disks to the DOS machine. Optionally, you can fill this directory with your floppy images now :).</p> Optional Access Control</h4> I recommend setting the setuid</a> bit as well so you can execute the binary without needing sudo. Since this is likely a private binary for your use alone, on a multiuser system (where you have sudo</code> privileges) you probably dont want others running ethflopd</code> either. I believe the following commands accomplish both:</p> Change the owner and group of ethflopd</code> to root</code> and your user's primary group</a> respectively. As far as I know, the owner must</em> be root so ethflopd</code> can create its runtime files and bind to an interface after setuid</code> takes effect. On the other hand, the group is your primary group, so you can still run the executable.</p> sudo chown root:wjones ~/.local/bin/ethflopd </code></pre> </li> Set the setuid</code> bit on ethflopd</code>. Make sure the group owner has executable permissions while the "others" group has the executable bit cleared. This ensures that only you or root can run ethflopd</code> if that's what you want.</em></p> sudo chmod 4754 ~/.local/bin/ethflopd </code></pre> </li> </ol> Installing ETHFLOP.COM</code></h3> At a minimum, you will need to transfer ETHFLOP.COM</code> and a packet driver for your particular NIC to your retro machine. If you can't find a packet driver for your card easily, tweet</a> or email</a> me and I'll see if I can help. I don't have a large</em> collection of packet drivers myself (CNET, Kingston, 3Com), but I can point you in the right direction.^{2</sup></a> </p>} Once you have ETHFLOP.COM</code> and a packet driver on your retro machine, you're good to go! There is no special installation required; ETHFLOP.COM</code> is self-contained. It's probably a good idea to make sure ETHFLOP.COM</code> and your packet driver can be seen by the DOS PATH</code> though.</p> Of course, one has to actually get</em> ETHFLOP.COM</code> from the distribution you unzipped on your Linux machine to the vintage machine in order to actually use</em> ethflop to its full potential. About that...</p> Transferring</em> ETHFLOP.COM</code> To The Target Machine</h4> Unfortunately, getting</em> files from a modern machine to a retro DOS machine is quite a topic in and of itself. There's various ways to do this- serial port, floppy, over the network, cdrom attached to the parallel port, zip disks, etc. Depending on how much extra hardware you have, transferring a file from a new to old machine ranges from fun to very tedious.</p> My Preferred Transfer Method- TCP/IP!</h5> It was fun for me personally to transfer ETHFLOP.COM</code> to an old machine because I had set up TCP/IP on my my PC AT years ago using mTCP</a>. mTCP applications integrate the TCP/IP stack into their binaries. The main distribution provides an FTP client (FTP</code>) and HTTP downloader (HTGET</code>), among other useful programs. I have also written my own</a> software using mTCP.</p> For transferring files from a new to old machine, I normally prefer using the FTP</code> client, but recently, I've had trouble setting up a temporary working FTP server using pyftpdlib</code></a>. Specifically, the server claims to transfer the file, but mTCP FTP</code> terminates the transfer immediately. Instead I decided to use python3</code>'s http.server</code> module and then download the file to my AT using HTGET</code>. Specifically:</p> On the Linux/modern side, invoke an HTTP server in the ethflop directory:</p> wjones@DietPi:~/src/ethflop$ python3 -m http.server Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ... </code></pre> </li> On the DOS/retro side, connect to the newly-invoked HTTP server and download ETHFLOP.COM</code> using HTGET</code>:</p> HTGET -o ETHFLOP.COM 192.168.1.162:8000/ethflop.com </code></pre> </li> </ol> In less than a second, I had ETHFLOP.COM</code> safely on my AT! The power of TCP/IP and Ethernet still amazes me.</p> I don't remember the exact details of how I got mTCP onto my AT initially, but as I recall I had to use a third computer with a zip drive and 5.25" floppy drive as an intermediate. I used a USB Zip drive to copy files from my laptop. The files were read by the intermediate computer's internal</em> Zip drive, and then copied onto the intermediate's 5.25" 1.2MB drive, which I then could write onto the AT's hard disk.</p> Bootstrapping</h5> I apologize in advance for the length of this section. This needs to be spun off into a blog post by itself. Feel free to skip</a> to the next section.</strong></p> In the absence of a working network connection, I understand using floppy disks for data transfer to old systems is sometimes not an option. While a floppy drive is pretty much the lowest common denominator of hardware for IBM PCs, clones, and descendants, I've seen a few issues:</p> Getting blank 5.25" floppies is inconvenient in 2019.</li> Even after you get a 5.25" drive for a second machine, I don't think any of the modern USB 5.25" floppy drive interfaces support writing disks (no demand?). So you need to setup up an intermediate machine as I described in the previous section.</li> Occassionally, I've seen people with broken floppy drives who want to install software onto their machine.</li> </ul> I think serial port is the method for transferring data between new and old machines that makes the least</em> amount of assumptions about available hardware. A serial port in DB-9 or DB-25</a> form factor is pretty a common hardware addition to even the oldest IBM PC and clones. Serial ports are traditionally accessed through the DOS command line by the special names COM[1-4]</code> or AUX</code> (for COMmmunications or AUXilary). Modern OSes have good support for DB-9 serial ports via USB adapters</a> or PCI Express</a> because RS-232 is still widely used for better or worse. You will need a null-modem</a> cable or adapter to connect your modern and retro computer together; these are still widely available to purchase.</p> There are a few bootstrap</a> mechanisms that have been developed over the years for DOS. They mostly involve typing in scripts (like BASIC) and/or small exectuables whose opcodes</a> or encoding</a> only use ASCII characters. Because the bootstraps only create files with ASCII characters, one can use the DOS COPY</code>command to read bootstrap programs from the serial port, such as COPY COM1 XFER.COM</code>. The ASCII restriction is because DOS uses the EOF</code> ASCII character (^Z</code>) to know when the transfer is finished.^{3</sup></a> </p>} Bootstrap programs often implement a minimal Kermit protocol</a> receiver program. Unlike the DOS COPY</code> command, the Kermit protocol can send and receive files with arbitrary bytes, and the protocol was designed to be extremely portable to new machines. The Kermit Project took the bootstrap problem of "transferring the file-transfer program to a new computer" seriously; all the bootstrap links I provided in this section are part of the Kermit archive at Columbia.</p> I use either minicom</code> or Tera Term</a> as my serial transfer programs on machines with modern OSes. Both of these programs understand the Kermit protocol and</em> can be used to send a file down the serial line as-is for the DOS COPY</code> command. However, you must manually send the EOF</code> (^Z</code>) control character yourself afterwards in both programs.</em></p> I can vouch personally that msbrcv.bas</code></a> has worked for me in the past with QBASIC</code>. I have not tested in older DOS versions using BASIC</code> or BASICA</code> as of this writing (12-8-2017). I plan to test transfers with TCOM.COM</code></a> in the near future, which may be preferable since it doesn't require BASIC. Really, I should probably write a blog post on bootstrapping and move this whole section there :).</p> Usage</h2> Starting and Stopping ethflopd</code></h3> To start the daemon, supply the following command line arguments in order:</p> The network interface you wish to listen on (eth0</code>, wlan0</code>- wireless interfaces work</em>, etc.) .</li> The storage directory of your floppy images.</li> </ol> My invocation looks like: ethflopd wlan0 ~/src/img/</code>. There is an optional -f</code> option to run the daemon in the foreground as well. I find this useful if you wish to invoke ethflopd</code> for as long as you're running your DOS machine, as opposed to running ethflopd</code> continuously.</p> Once ethflopd</code> is running, it will listen for the MAC address</a> of a networked DOS computer running ETHFLOP.COM</code> to send and receive virtual floppy images. From the DOS side, detecting the MAC address of a machine running ethflopd</code> is automatic. To reiterate, there is no IP address setup because ethflop uses raw Ethernet/Wifi frames to transfer data.</p> To stop the daemon, you need to kill</code> the PID associated with ethflopd</code> or run pkill ethflopd</code>. sudo</code> may be required depending on your particular setup. If ethflop</code> is running in the foreground, ^C</code> works fine as well :); ethflopd</code> has the same cleanup code for both kill</code> and ^C</code>.</p> A Guided Tour Of ETHFLOP.COM</code></h3> Once the daemon is running on the Linux machine, the first thing to do on the DOS end is... read the help text :). I have copied the output of running ethflop without any arguments for your convenience!</p> C:\>ethflop ethflop v0.6 - a floppy drive emulator over Ethernet Copyright (C) 2019 Mateusz Viste === USAGE ==================================================================== ethflop a installs the ethflop TSR as A: ethflop b installs ethflop as B: (works only if you have a real B:) ethflop i DISKNAME 'inserts' the virtual floppy named 'DISKNAME' ethflop ip DSKNAME same as 'i', but the inserted floppy is WRITE PROTECTED ethflop r OLD NEW renames virt. floppy 'OLD' to 'NEW' ethflop e 'ejects' currently loaded virtual floppy ethflop nSZ DSKNAME creates a new virt. floppy DSKNAME, SZ KB big. SZ can be: 360, 720, 1200, 1440, 2880, 4800, 8100, 9600, 15500, 31000 ethflop l displays the list of available virt. floppies ethflop d DISKNAME DELETES virt. floppy named DISKNAME - BE CAREFUL! ethflop s displays current status of the ethflop TSR ethflop u unloads the ethflop TSR === NOTES ==================================================================== * Disk names must be 1 to 8 characters long. Only A-Z, 0-9 and '_-' allowed. * ethflop requires the presence of an ethflop server on the local network. === LICENSE ================================================================== ethflop is published under the terms of the ISC license. See ETHFLOP.TXT. </code></pre> ethflop can basically be divided into three command types:</p> Load/unload TSR (a</code>, b</code>, u</code>)</li> Query commands (l</code>, s</code>)</li> Operate on virtual disks (i</code>, ip</code>, r</code>, e</code>, n</code>, d</code>)</li> </ul> Loading/Unloading the TSR</h4> Packet Driver Installation</h5> The first</del> second</ins> thing to do is to make sure your packet driver is loaded into memory. In my case with the 3c509, this looks like:</p> C:\>3c5x9pd 0x60 3Com EtherLink 10 ISA Packet Driver v1.2 (C) Copyright 1993 3Com Corp. All rights reserved. Error: The interrupt requested is already in use by another packet driver. </code></pre> The lone argument to 3C5X9PD.COM</code> is the x86 interrupt number to use to call into the packet driver. Interrupt 0x60</code> is a good traditional default, as indicated by the error message telling me I already loaded the driver.</p> Some packet drivers require more options to set up, but chances are the default options provided in your NIC manual or by the supplied configuration program- as is the case with the 3c509- will work correctly. Older machines may require a bit more fine tuning, but between three different NIC manufactuters over a dozen DOS machines, I've had the default options not work exactly once^{4</sup></a> . Manually assigning resources becomes more important if you have multiple NICs installed in a retro machine, but this is rare.</p>} TSR Installation</h5> After you're sure a packet driver is loaded, then it's time to load the TSR! As I said before, if you only have a single physical floppy drive, you must use ethflop a</code>. However, seeing as I do</em> have a physical B:\ drive, I installed ethflop at drive B:\:</p> C:\>ethflop b server found at F0:03:8C:90:B2:A1 current virt. floppy: <NONE> ethflop has been installed </code></pre> ETHFLOP.COM</code> will attempt to automatically find</a> a server running ethflopd</code> without user invervention. It can</em> take multiple tries to load the driver depending on your network connection quality. However, I've found that once ETHFLOP.COM</code> finds</em> the server, any subsequent network errors are transient and the TSR keeps working.</p> Once ethflop is installed, you lose access to the underlying physical drive. This also means that you can't use ethflop if you don't have a physical floppy drive.</em>^{5</sup></a> The virtual floppy drive presented by ethflop will be empty; you must either load or create a virtual disk before accessing the drive, or else you will get DOS' famous error message:</p>} C:\>B: Seek error reading drive B Abort, Retry, Fail?f Current drive is no longer valid>C: C:\> </code></pre> I have not tested whether ethflop can coexist with applications using a DOS TCP/IP stack, though I can't see any immediate problems.</p> Unloading ETHFLOP.COM</code></h5> When you're done playing with virtual floppies and/or need to access the physical drive, type ethflop u</code> to uninstall the TSR:</p> C:\>ethflop u ethflop has been uninstalled </code></pre> You will get access back to your underlying physical floppy drive as well as the NIC after unloading.</p> Querying commands</h4> You can get information about the ethflop TSR and what disks are available by running ethflop s</code> and ethflop l</code> respectively:</p> C:\>ethflop s ethflop is currently installed as drive B: server is at F0:03:8C:90:B2:A1 current virt. floppy: <NONE> C:\>ethflop l msdos213 mydisk </code></pre> Uh oh, one of my virtual disks is missing from the list! As it turns out, the file extension for your images matter</em>! I've been sloppy about this in the past, switching between .ima</code> or .img</code> on a whim. Renaming mtcpget.ima</code> to mtcpget.img</code> in my image directory on the Tinker Board makes all three images appear during a query:</p> C:\>ethflop l msdos213 mtcpget mydisk </code></pre> Much better!</p> Using Virtual Disks</h4> Commands to operate on virtual disks are the bulk of ETHFLOP.COM</code>'s functionality. Any typical action you would perform on a physical floppy disk has an equivalent coded into ETHFLOP.COM</code>:</p> i</code>- Insert a virtual disk (place a disk into the disk drive).</li> e</code>- Eject a virtual disk (remove the disk from the disk drive).</li> ip</code>- Write-protect and insert a virtual disk (cover a notch or slide a switch on the disk, then insert).</li> r</code>- Rename a virtual disk (change the label on the sleeve/cover).</li> n</code>- Create a new virtual disk (go to the store and buy some floppies).</li> d</code>- Delete a virutal disk (erase a good floppy or discard a bad floppy).</li> </ul> I present these commands in the order I tried them out in my initial ETHFLOP session, which matches the list above.</p> Insert</h5> Let's insert a disk! I chose mtcpget</code> because I actually forgot the contents of this image and wanted to see what I used it for. Listing the root directory using DIR</code> works without a hitch:</p> C:\>ethflop i mtcpget Disk MTCPGET loaded (360 KiB) C:\>B: B:\>dir Volume in drive B has no label Volume Serial Number is AEA9-1EC3 Directory of B:\ MTCPGET BAT 870 01-01-80 3:34p PKT_DRVS <DIR> 05-16-12 1:52p UTILS <DIR> 05-16-12 1:53p CFG <DIR> 05-16-12 1:53p BATCH <DIR> 05-16-12 2:15p GO BAT 0 05-16-12 2:15p 6 file(s) 870 bytes 101376 bytes free B:\> </code></pre> Looks like I used this disk to bootstrap mTCP onto new machines. Displaying the contents of the MTCPGET.BAT</code> file using TYPE</code> confirms this. Good to know that accessing files works fine as well.</p> B:\>type MTCPGET.BAT REM ECHO OFF ECHO This batch file retrieves the mTCP stack and programs using mTCP FTP. ECHO Make sure environment is unset and batch variables are set correctly before using this file. ECHO In addition, make sure the packet driver is functional. PAUSE SET SRCDRIV=A: SET PATH=%SRCDRIV%\UTILS SET DESTDRIV=C: SET MACHINE=8088 SET MTCPCFG=%SRCDRIV%\CFG\BATCHTCP.CFG SET SERVER=192.168.1.3 ECHO Switching to destination drive... %DESTDRIV% ECHO Obtaining DHCP configuration... DHCP ECHO Obtaining mTCP stack/programs... FTP %SERVER% < %SRCDRIV%\CFG\MTCP.FTP ECHO Obtaining machine-specific files... FTP %SERVER% < %SRCDRIV%\CFG\%MACHINE%.FTP ECHO Switching back to source drive... %SRCDRIV% ECHO This batch file has completed executing. ECHO It is recommended to restart the computer at this time to restore the environment. PAUSE </code></pre> Eject</h5> I didn't have much to do with MTCPGET</code>, so I soon ejected this disk. I tested that this worked by deliberately accessing a non-existent disk. I am curious that f</code>ailing</a> on the first error successfully read out a blank volume label, but even with that behavior, the eject command works fine to me:</p> B:\>ethflop e Disk MTCPGET ejected B:\>dir Seek error reading drive B Abort, Retry, Fail?f Volume in drive B has no label Seek error reading drive B Abort, Retry, Fail?f Not ready reading drive B Abort, Retry, Fail?f Fail on INT 24 Seek error reading drive B Abort, Retry, Fail?f Current drive is no longer valid>C: </code></pre> Write Protect, Then Insert</h5> Because I live life on the edge, I decided to test whether write-protect works on one of my pristine, old MS-DOS images. If this fails, then I've lost... pretty much nothing, because this is a backup copy of the image :). I don't know why the volume label is missing 3 hex digits- I will investigate and update later.</p> C:\>ethflop ip msdos213 Disk MSDOS213 loaded (360 KiB (write-protected)) C:\>B: B:\>mkdir FOO Write protect error writing drive B Abort, Retry, Fail?f Fail on INT 24 - FOO B:\>dir Volume in drive B is E107-A Directory of B:\ BIN <DIR> 08-27-11 12:51p COMMAND COM 16421 07-16-84 9:50a ALTCHAR SYS 431 04-04-84 8:22a AUTOEXEC BAT 23 04-04-84 8:11a 4 file(s) 16875 bytes 75776 bytes free B:\> </code></pre> Since write-protected succeeded according to the error, it looks like I saved myself an extra rsync</code>/cp</code> command to reset my MS-DOS image. And based on the modification date of BIN</code>, this image isn't pristine/original at all! Ah well...</p> Rename</h5> Renaming a virtual disk has interesting behavior- the rename operation appears to be case insensitive, even on the Linux side! I expected the image filename on the server to be renamed with the same case that I typed it at the DOS prompt:</p> B:\>ethflop r mydisk MYDBAK Disk MYDISK renamed to MYDBAK </code></pre> However, it looks like either ethflopd</code> or ETHFLOP.COM</code> sanitizes the image name so that only lowercase image filenames are created on the server side:</p> wjones@DietPi:~/src/img$ ls msdos213.img mtcpget.img mydbak.img </code></pre> I did not try renaming a virtual disk that's inserted. I will update this post in the future with the behavior when I have time to test.</p> MYDISK</code> was an image I had created in a prior ethflop session. I renamed it to MYDBAK</code> (BAK</code> as in "backup") because I want to duplicate the process of creating MYDISK</code> with a fresh new image for reasons I will describe in the next section</em>. Speaking of which...</p> Create An Image</h5> The most powerful feature of ethflop in my opinion is the ability to create new virtual disks to store on the server. Compared to copying floppies, creating and populating a virtual disk is much more convenient to quickly share a new disk with your other networked computers. You can even store your shiny new image image on an FTP server or BBS (those things still exist, right?) to share with other like-minded retro-enthusiasts in record time!</p> To create an image, use the n</code> command and supply a size and disk name as arguments:</p> B:\>ethflop n360 mydisk Disk MYDISK created (360 KiB) </code></pre> Listing the image directory contents shows that I indeed create a new floppy image. Note that the image owner and group match those of the user invoking ethflopd</code> (after setuid</code> takes effect). The permissions of mydbak.img</code> in the listing were from a previous ethflopd</code> session using sudo</code>:</p> wjones@DietPi:~/src/img$ ls -l total 848 -rw-r--r-- 1 wjones wjones 368640 Sep 22 2012 msdos213.img -rw-r--r-- 1 wjones wjones 368640 May 16 2012 mtcpget.img -rw-r--r-- 1 root root 368640 Dec 4 13:53 mydbak.img -rw-r--r-- 1 root wjones 368640 Dec 5 13:19 mydisk.img </code></pre> After creating an image, you need to insert your shiny new disk before using it. Based on the Seek error</code> messages, I was having network trouble at the time. However, I did</em> eventually manage to connect to the server:</p> B:\>ethflop i mydisk ERROR: you must first eject your current virtual floppy (MSDOS213) B:\>ethflop e Disk MSDOS213 ejected B:\>ethflop i mydisk Seek error reading drive B Abort, Retry, Fail?r Seek error reading drive B Abort, Retry, Fail?r Seek error reading drive B Abort, Retry, Fail?r Seek error reading drive B Abort, Retry, Fail?f Invalid drive specification Disk MYDISK loaded (360 KiB) B:\>dir Volume in drive B has no label Volume Serial Number is 5DE9-4A1E Directory of B:\ File not found B:\> </code></pre> Once ETHFLOP.COM</code> connected to the server, I didn't have any further trouble.</p> On a lark, I decided to try to recreate my existing MYDBAK</code> virtual disk, including the TEST</code> directory and TEST.TXT</code> file. I was curious what bytes, if any, would be different between the two images beyond timestamps. Unfortunately, I made a mistake and accidentally quoted the contents of TEST.TXT</code>. My mistake will come up again</a> a bit later.</p> B:\>mkdir TEST Not ready writing drive B Abort, Retry, Fail?r B:\>cd TEST B:\TEST>echo "This is a test file" > TEST.TXT B:\TEST>ethflop e Disk MYDISK ejected </code></pre> Unsupported Hardware Floppy Density Emulation</h6> You can even create images of floppy densities that the hardware doesn't support! In the below snippet, I create a 1.44MB 3.5" virtual disk and populate it much the same way as the previous examples, minus the quotes this time :):</p> C:\>ethflop n1440 hddisk Disk HDDISK created (1440 KiB) C:\>ethflop i hddisk Disk HDDISK loaded (1440 KiB) C:\>B: B:\>mkdir TEST B:\>dir Volume in drive B has no label Volume Serial Number is 5DE9-4BEB Directory of B:\ TEST <DIR> 12-05-19 1:27p 1 file(s) 0 bytes 1457152 bytes free B:\TEST>echo This is a test file > TEST.TXT B:\TEST>type TEST.TXT This is a test file B:\TEST>ethflop e Disk HDDISK ejected </code></pre> By doing a hexdump on the server (alias hd='od -Ax -t x1z'</code>), you can see the image indeed is the size of a 1.44MB disk, and includes the file I just created starting at 004400</code>. File metadata- known in FAT file system terms as the directory entry</a>- is also visible starting at 004240</code>:</p> wjones@DietPi:~/src/img$ hd hddisk.img 000000 eb fe 90 4d 53 44 4f 53 35 2e 30 00 02 01 01 00 >...MSDOS5.0.....< 000010 02 e0 00 40 0b f0 09 00 12 00 02 00 00 00 00 00 >...@............< 000020 00 00 00 00 00 00 29 eb 4b e9 5d 4e 4f 20 4e 41 >......).K.]NO NA< 000030 4d 45 20 20 20 20 46 41 54 31 32 20 20 20 00 00 >ME FAT12 ..< 000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 0001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa >..............U.< 000200 f0 ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001400 f0 ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 001410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 002600 54 45 53 54 20 20 20 20 20 20 20 10 00 00 00 00 >TEST .....< 002610 00 00 00 00 00 00 62 6b 85 4f 02 00 00 00 00 00 >......bk.O......< 002620 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 004200 2e 20 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >. .....< 004210 00 00 00 00 00 00 62 6b 85 4f 02 00 00 00 00 00 >......bk.O......< 004220 2e 2e 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >.. .....< 004230 00 00 00 00 00 00 62 6b 85 4f 00 00 00 00 00 00 >......bk.O......< 004240 54 45 53 54 20 20 20 20 54 58 54 20 00 00 00 00 >TEST TXT ....< 004250 00 00 00 00 00 00 aa 6b 85 4f 03 00 16 00 00 00 >.......k.O......< 004260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 004400 54 68 69 73 20 69 73 20 61 20 74 65 73 74 20 66 >This is a test f< 004410 69 6c 65 20 0d 0a 00 60 a9 16 eb 00 02 2f 00 00 >ile ...`...../..< 004420 46 44 49 53 4b 20 20 20 43 4f 4d 00 00 00 00 00 >FDISK COM.....< 004430 00 00 00 00 00 00 00 60 a9 16 f1 00 d8 df 00 00 >.......`........< 004440 46 4f 52 4d 41 54 20 20 43 4f 4d 00 00 00 00 00 >FORMAT COM.....< 004450 00 00 00 00 00 00 00 60 a9 16 0d 01 8f 80 00 00 >.......`........< 004460 4d 49 52 52 4f 52 20 20 43 4f 4d 00 00 00 00 00 >MIRROR COM.....< 004470 00 00 00 00 00 00 00 60 a9 16 1e 01 f9 46 00 00 >.......`.....F..< 004480 33 43 35 58 39 50 44 20 43 4f 4d 00 00 00 00 00 >3C5X9PD COM.....< 004490 00 00 00 00 00 00 ac 80 85 26 89 13 bc 31 00 00 >.........&...1..< 0044a0 53 48 41 52 45 20 20 20 45 58 45 00 00 00 00 00 >SHARE EXE.....< 0044b0 00 00 00 00 00 00 00 60 a9 16 2a 01 90 2a 00 00 >.......`......< 0044c0 53 4d 41 52 54 44 52 56 53 59 53 00 00 00 00 00 >SMARTDRVSYS.....< 0044d0 00 00 00 00 00 00 00 60 a9 16 30 01 83 20 00 00 >.......`..0.. ..< 0044e0 53 59 53 20 20 20 20 20 43 4f 4d 00 00 00 00 00 >SYS COM.....< 0044f0 00 00 00 00 00 00 00 60 a9 16 35 01 70 34 00 00 >.......`..5.p4..< 004500 55 4e 44 45 4c 45 54 45 45 58 45 00 00 00 00 00 >UNDELETEEXE.....< 004510 00 00 00 00 00 00 00 60 a9 16 3c 01 64 36 00 00 >.......`..<.d6..< 004520 55 4e 46 4f 52 4d 41 54 43 4f 4d 00 00 00 00 00 >UNFORMATCOM.....< 004530 00 00 00 00 00 00 00 60 a9 16 43 01 90 48 00 00 >.......`..C..H..< 004540 58 43 4f 50 59 20 20 20 45 58 45 00 00 00 00 00 >XCOPY EXE.....< 004550 00 00 00 00 00 00 00 60 a9 16 4d 01 bc 3d 00 00 >.......`..M..=..< 004560 44 4f 53 4b 45 59 20 20 43 4f 4d 00 00 00 00 00 >DOSKEY COM.....< 004570 00 00 00 00 00 00 00 60 a9 16 55 01 fc 16 00 00 >.......`..U.....< 004580 44 4f 53 53 48 45 4c 4c 56 49 44 00 00 00 00 00 >DOSSHELLVID.....< 004590 00 00 00 00 00 00 00 60 a9 16 58 01 f6 24 00 00 >.......`..X..$..< 0045a0 54 43 50 20 20 20 20 20 20 20 20 10 00 00 00 00 >TCP .....< 0045b0 00 00 00 00 00 00 ee 10 24 c8 aa 04 00 00 00 00 >........$.......< 0045c0 44 4f 53 53 48 45 4c 4c 43 4f 4d 00 00 00 00 00 >DOSSHELLCOM.....< 0045d0 00 00 00 00 00 00 00 60 a9 16 63 01 02 12 00 00 >.......`..c.....< 0045e0 44 4f 53 53 48 45 4c 4c 45 58 45 00 00 00 00 00 >DOSSHELLEXE.....< 0045f0 00 00 00 00 00 00 00 60 a9 16 66 01 0e 98 03 00 >.......`..f.....< 004600 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 168000 </code></pre> I assume the density of floppy images you can create and write depends on the DOS version you're running. My 286 runs PC-DOS 5.0, which supports 1.44MB disks just fine, but I know my 286 BIOS does not</em> support 1.44MB disks. I don't know at present how ETHFLOP.COM</code> interacts with the BIOS bookkeeping, if at all. Until I know for sure, I am using this feature with caution.</strong></p> In the above dump of hddisk.img</code>, the data starting at FDISK</code> and beyond appear to be legitimate directory entries from my AT's hard disk. I am not sure why ethflopd</code> is creating these entries.</em> My guess is this behavior isn't intentional, as none of these directory entries are actually reachable; bytes 002620</code> and 004260</code> would need to be nonzero</a> to indicate more files are present in either the root or TEST</code> directories.</p> Seeing as I'm not storing sensitive data on my AT, I'm not exactly worried about extra data being copied to my newly-created virtual disks. However, I found the behavior strange enough to be worth noting. If you're a retro enthusiast who stores sensitive data on their retro machines (I'm</em> not gonna judge you), I would check to make sure newly-created virtual images don't accidentally copy sensitive data to the server. In the meantime, I'll look further into this behavior.</p> Unexpected Directory Entries</h6> What follows is my attempts to flesh out why extra directory entries are created. If you don't feel like looking at hexdumps, skip</a> to the next section!</p> The extra directory entries seem to disappear when I recreate</em> an existing image. For example, my second (and still wrong</em>) attempt to duplicate mydbak.img</code> from a fresh copy of mydisk.img</code> resulted in the following hexdumps:</p> wjones@DietPi:~/src/img$ hd mydisk.img 000000 eb fe 90 4d 53 44 4f 53 35 2e 30 00 02 02 01 00 >...MSDOS5.0.....< 000010 02 70 00 d0 02 fd 02 00 09 00 02 00 00 00 00 00 >.p..............< 000020 00 00 00 00 00 00 29 1e 4a e9 5d 4e 4f 20 4e 41 >......).J.]NO NA< 000030 4d 45 20 20 20 20 46 41 54 31 32 20 20 20 00 00 >ME FAT12 ..< 000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 0001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa >..............U.< 000200 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000600 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000610 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000a00 54 45 53 54 20 20 20 20 20 20 20 10 00 00 00 00 >TEST .....< 000a10 00 00 00 00 00 00 b7 6a 85 4f 02 00 00 00 00 00 >.......j.O......< 000a20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001800 2e 20 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >. .....< 001810 00 00 00 00 00 00 b7 6a 85 4f 02 00 00 00 00 00 >.......j.O......< 001820 2e 2e 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >.. .....< 001830 00 00 00 00 00 00 b7 6a 85 4f 00 00 00 00 00 00 >.......j.O......< 001840 54 45 53 54 20 20 20 20 54 58 54 20 00 00 00 00 >TEST TXT ....< 001850 00 00 00 00 00 00 c9 6a 85 4f 03 00 18 00 00 00 >.......j.O......< 001860 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001c00 22 54 68 69 73 20 69 73 20 61 20 74 65 73 74 20 >"This is a test < 001c10 66 69 6c 65 22 20 0d 0a 00 00 00 00 00 00 00 00 >file" ..........< 001c20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 05a000 wjones@DietPi:~/src/img$ hd mydbak.img 000000 eb fe 90 4d 53 44 4f 53 35 2e 30 00 02 02 01 00 >...MSDOS5.0.....< 000010 02 70 00 d0 02 fd 02 00 09 00 02 00 00 00 00 00 >.p..............< 000020 00 00 00 00 00 00 29 7c 00 e8 5d 4e 4f 20 4e 41 >......)\|..]NO NA< 000030 4d 45 20 20 20 20 46 41 54 31 32 20 20 20 00 00 >ME FAT12 ..< 000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 0001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa >..............U.< 000200 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000600 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000610 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000a00 54 45 53 54 20 20 20 20 20 20 20 10 00 00 00 00 >TEST .....< 000a10 00 00 00 00 00 00 9d 6e 84 4f 02 00 00 00 00 00 >.......n.O......< 000a20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001800 2e 20 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >. .....< 001810 00 00 00 00 00 00 9d 6e 84 4f 02 00 00 00 00 00 >.......n.O......< 001820 2e 2e 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >.. .....< 001830 00 00 00 00 00 00 9d 6e 84 4f 00 00 00 00 00 00 >.......n.O......< 001840 54 45 53 54 20 20 20 20 54 58 54 20 00 00 00 00 >TEST TXT ....< 001850 00 00 00 00 00 00 b0 6e 84 4f 03 00 16 00 00 00 >.......n.O......< 001860 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001c00 54 68 69 73 20 69 73 20 61 20 74 65 73 74 20 66 >This is a test f< 001c10 69 6c 65 2e 0d 0a 00 60 a9 16 eb 00 02 2f 00 00 >ile....`...../..< 001c20 46 44 49 53 4b 20 20 20 43 4f 4d 00 00 00 00 00 >FDISK COM.....< 001c30 00 00 00 00 00 00 00 60 a9 16 f1 00 d8 df 00 00 >.......`........< 001c40 46 4f 52 4d 41 54 20 20 43 4f 4d 00 00 00 00 00 >FORMAT COM.....< 001c50 00 00 00 00 00 00 00 60 a9 16 0d 01 8f 80 00 00 >.......`........< 001c60 4d 49 52 52 4f 52 20 20 43 4f 4d 00 00 00 00 00 >MIRROR COM.....< 001c70 00 00 00 00 00 00 00 60 a9 16 1e 01 f9 46 00 00 >.......`.....F..< 001c80 33 43 35 58 39 50 44 20 43 4f 4d 00 00 00 00 00 >3C5X9PD COM.....< 001c90 00 00 00 00 00 00 ac 80 85 26 89 13 bc 31 00 00 >.........&...1..< 001ca0 53 48 41 52 45 20 20 20 45 58 45 00 00 00 00 00 >SHARE EXE.....< 001cb0 00 00 00 00 00 00 00 60 a9 16 2a 01 90 2a 00 00 >.......`......< 001cc0 53 4d 41 52 54 44 52 56 53 59 53 00 00 00 00 00 >SMARTDRVSYS.....< 001cd0 00 00 00 00 00 00 00 60 a9 16 30 01 83 20 00 00 >.......`..0.. ..< 001ce0 53 59 53 20 20 20 20 20 43 4f 4d 00 00 00 00 00 >SYS COM.....< 001cf0 00 00 00 00 00 00 00 60 a9 16 35 01 70 34 00 00 >.......`..5.p4..< 001d00 55 4e 44 45 4c 45 54 45 45 58 45 00 00 00 00 00 >UNDELETEEXE.....< 001d10 00 00 00 00 00 00 00 60 a9 16 3c 01 64 36 00 00 >.......`..<.d6..< 001d20 55 4e 46 4f 52 4d 41 54 43 4f 4d 00 00 00 00 00 >UNFORMATCOM.....< 001d30 00 00 00 00 00 00 00 60 a9 16 43 01 90 48 00 00 >.......`..C..H..< 001d40 58 43 4f 50 59 20 20 20 45 58 45 00 00 00 00 00 >XCOPY EXE.....< 001d50 00 00 00 00 00 00 00 60 a9 16 4d 01 bc 3d 00 00 >.......`..M..=..< 001d60 44 4f 53 4b 45 59 20 20 43 4f 4d 00 00 00 00 00 >DOSKEY COM.....< 001d70 00 00 00 00 00 00 00 60 a9 16 55 01 fc 16 00 00 >.......`..U.....< 001d80 44 4f 53 53 48 45 4c 4c 56 49 44 00 00 00 00 00 >DOSSHELLVID.....< 001d90 00 00 00 00 00 00 00 60 a9 16 58 01 f6 24 00 00 >.......`..X..$..< 001da0 54 43 50 20 20 20 20 20 20 20 20 10 00 00 00 00 >TCP .....< 001db0 00 00 00 00 00 00 ee 10 24 c8 aa 04 00 00 00 00 >........$.......< 001dc0 44 4f 53 53 48 45 4c 4c 43 4f 4d 00 00 00 00 00 >DOSSHELLCOM.....< 001dd0 00 00 00 00 00 00 00 60 a9 16 63 01 02 12 00 00 >.......`..c.....< 001de0 44 4f 53 53 48 45 4c 4c 45 58 45 00 00 00 00 00 >DOSSHELLEXE.....< 001df0 00 00 00 00 00 00 00 60 a9 16 66 01 0e 98 03 00 >.......`..f.....< 001e00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 05a000 </code></pre> mydbak.img</code> was created during a previous session and contains directory entries from my AT hard disk, while mydisk.img</code> does not. When I recreate mydisk.img</code> yet again</em>, I see the same results- no directory entries.</p> It was at this point I realized I accidentally included quotes in TEST.TXT</code> while duplicating the layout of mydbak.img</code>. On a hunch, I deleted</a> mydisk.img</code> completely, created a new disk, and created TEST</code> and TEST.TXT</code> correctly</em> this time. The hexdump of my new mydisk.img</code> looks like this:</p> wjones@DietPi:~/src/img$ hd mydisk.img 000000 eb fe 90 4d 53 44 4f 53 35 2e 30 00 02 02 01 00 >...MSDOS5.0.....< 000010 02 70 00 d0 02 fd 02 00 09 00 02 00 00 00 00 00 >.p..............< 000020 00 00 00 00 00 00 29 1e 4a e9 5d 4e 4f 20 4e 41 >......).J.]NO NA< 000030 4d 45 20 20 20 20 46 41 54 31 32 20 20 20 00 00 >ME FAT12 ..< 000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 0001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa >..............U.< 000200 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000600 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000610 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000a00 54 45 53 54 20 20 20 20 20 20 20 10 00 00 00 00 >TEST .....< 000a10 00 00 00 00 00 00 b7 6a 85 4f 02 00 00 00 00 00 >.......j.O......< 000a20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001800 2e 20 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >. .....< 001810 00 00 00 00 00 00 b7 6a 85 4f 02 00 00 00 00 00 >.......j.O......< 001820 2e 2e 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >.. .....< 001830 00 00 00 00 00 00 b7 6a 85 4f 00 00 00 00 00 00 >.......j.O......< 001840 54 45 53 54 20 20 20 20 54 58 54 20 00 00 00 00 >TEST TXT ....< 001850 00 00 00 00 00 00 10 6c 85 4f 03 00 17 00 00 00 >.......l.O......< 001860 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001c00 54 68 69 73 20 69 73 20 61 20 74 65 73 74 20 66 >This is a test f< 001c10 69 6c 65 2e 20 0d 0a 0e 24 c8 02 00 00 00 00 00 >ile. ...$.......< 001c20 2e 2e 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >.. .....< 001c30 00 00 00 00 00 00 68 0e 24 c8 00 00 00 00 00 00 >......h.$.......< 001c40 43 4f 55 4e 54 52 59 20 53 59 53 00 00 00 00 00 >COUNTRY SYS.....< 001c50 00 00 00 00 00 00 00 60 a9 16 3f 00 16 45 00 00 >.......`..?..E..< 001c60 45 47 41 20 20 20 20 20 53 59 53 00 00 00 00 00 >EGA SYS.....< 001c70 00 00 00 00 00 00 00 60 a9 16 48 00 15 13 00 00 >.......`..H.....< 001c80 4b 45 59 42 20 20 20 20 43 4f 4d 00 00 00 00 00 >KEYB COM.....< 001c90 00 00 00 00 00 00 00 60 a9 16 4b 00 3b 3b 00 00 >.......`..K.;;..< 001ca0 4b 45 59 42 4f 41 52 44 53 59 53 00 00 00 00 00 >KEYBOARDSYS.....< 001cb0 00 00 00 00 00 00 00 60 a9 16 53 00 3e 95 00 00 >.......`..S.>...< 001cc0 4e 4c 53 46 55 4e 43 20 45 58 45 00 00 00 00 00 >NLSFUNC EXE.....< 001cd0 00 00 00 00 00 00 00 60 a9 16 66 00 6c 1b 00 00 >.......`..f.l...< 001ce0 44 49 53 50 4c 41 59 20 53 59 53 00 00 00 00 00 >DISPLAY SYS.....< 001cf0 00 00 00 00 00 00 00 60 a9 16 6a 00 a5 3d 00 00 >.......`..j..=..< 001d00 45 47 41 20 20 20 20 20 43 50 49 00 00 00 00 00 >EGA CPI.....< 001d10 00 00 00 00 00 00 00 60 a9 16 72 00 e0 e5 00 00 >.......`..r.....< 001d20 48 49 4d 45 4d 20 20 20 53 59 53 00 00 00 00 00 >HIMEM SYS.....< 001d30 00 00 00 00 00 00 00 60 a9 16 8f 00 20 2d 00 00 >.......`.... -..< 001d40 4d 4f 44 45 20 20 20 20 43 4f 4d 00 00 00 00 00 >MODE COM.....< 001d50 00 00 00 00 00 00 00 60 a9 16 95 00 21 5c 00 00 >.......`....!\..< 001d60 53 45 54 56 45 52 20 20 45 58 45 00 00 00 00 00 >SETVER EXE.....< 001d70 00 00 00 00 00 00 00 60 a9 16 a1 00 e1 2e 00 00 >.......`........< 001d80 41 4e 53 49 20 20 20 20 53 59 53 00 00 00 00 00 >ANSI SYS.....< 001d90 00 00 00 00 00 00 00 60 a9 16 a7 00 3a 23 00 00 >.......`....:#..< 001da0 44 45 42 55 47 20 20 20 43 4f 4d 00 00 00 00 00 >DEBUG COM.....< 001db0 00 00 00 00 00 00 00 60 a9 16 ac 00 8a 50 00 00 >.......`.....P..< 001dc0 45 44 4c 49 4e 20 20 20 43 4f 4d 00 00 00 00 00 >EDLIN COM.....< 001dd0 00 00 00 00 00 00 00 60 a9 16 b7 00 52 31 00 00 >.......`....R1..< 001de0 45 4d 4d 33 38 36 20 20 45 58 45 00 00 00 00 00 >EMM386 EXE.....< 001df0 00 00 00 00 00 00 00 60 a9 16 be 00 5e 66 01 00 >.......`....^f..< 001e00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 05a000 </code></pre> I see directory entries yet again, and they're not even the same ones present in mydbak.img</code>!</em> Now I'm confident this behavior isn't intentional!</p> Delete</h5> Because I'm cautious about creating virtual disks of densities/sizes that my hardware doesn't support, I decided to delete HDDISK</code>/hddisk</code> to remove the temptation:</p> C:\>ethflop d hddisk Disk HDDISK has been deleted </code></pre> On the server side, there are no surprises- hddisk.img</code> is gone. If someone really</em> wants it back, they can recreate it from the hexdump!</p> wjones@DietPi:~/src/img$ ls msdos213.img mtcpget.img mydbak.img mydisk.img </code></pre> At this point, I had tested ethflop's functionality to my satisfaction and turned off the AT for the day. I'm sure I'll be using both of them again very soon.</p> The Verdict</h2> ethflop works pretty much flawlessly for accessing my myriad of floppy images I've accumulated over the years. As if you couldn't guess, I love seeing old hardware given a new lease on life. This applies especially when new software extends</em> the functionality of old hardware to- or beyond- what it was capable of during its useful life. So to Mateusz Viste</a>, I sincerely thank you for writing ethflop!</p> I have a large number of floppy images I've accumulated over the years, but not enough 5.25" floppies^{6</sup></a> to write them all. In the past I've reused disks I've no longer needed (but had an image backup) in order to "time division multiplex" floppy disks. But I have plenty of images which only work properly as boot disks. Since I can't load ethflop during boot, I like to save my physical floppies for stuff like install or rescue disks. I think I will be using ethflop frequently from here on out for my vintage pursuits, unless I'm just in the mood to play with real</em>, physical floppy disks :).</p>} Future Ideas</h3> I'm not in the right frame of mind to try this now, but I wonder if it'd be possible to extend ethflop to emulate a virtual D:\ drive (I dare not try replacing drive C:\!)? Or better yet, is it possible to extend ethflop into an MSDOS block device driver instead of a TSR? What about systems with 4 floppy drives? They're rare, but the original IBM PC disk controller supports them in principle</em>^{7</sup></a> . Oh, and I think I can even use ethflop to implement}DISKCOPY</code></a> over the network for unimaged floppies!</p> Mateusz Viste also wrote a networked file system</a> for DOS that I want to check out soon- that also might give me some ideas. There are many neat DOS and floppy project ideas still waiting to be pulled from the ether to give old machines new life...</p> Footnotes</h2> 1</a> I've opened my 10BASE2 to 10BASET converter before; the conversion is done by a chip that was manufactured by National Semiconductor in the early/mid-90s. Unfortunately, its name escapes me, and I can't check in the immediate future.</p> 2</a> I'm unsure if a "generic" packet driver exists, though I'm aware that the packet driver spec maintainer Russ Nelson</a> used to have a generic template that vendors used to create their packet drivers.</p> 3</a> Interfacing DOS with the serial port could be a blog post in and of itself, I'm afraid. It's very easy to get a very helpful Write failure</code> or General failure</code> when accessing the serial port (e.g. COPY COM2 FOO.TXT</code> or CTTY COM2</code>) just by breathing wrong. What best</em> matches my experience is that DOS uses flow control</a> for sending but not receiving (Source</a>). Disabling flow control in your serial terminal or transfer program is a safe option.</p> 4</a> I don't remember the exact details right now (and will check later), but I had an old 8-bit ISA CNet NIC that required a few kB of Upper Memory</a>. The manufacturer-set default switches for the memory area- yes, this had to be set manually!- conflicted with the memory area commonly used for hard drive controller (segment 0xc800</code>). This was even mentioned in the manual I got with the card, but of course I didn't read it first :).</p> 5</a> Originally, the IBM PC had provisions for 4 floppy drives, though I don't remember how DOS assigns drive letters in this case, especially in the case where hard disks exist as well. This might be an interesting addition to ethflop to keep Drives A:\ and B:\ free.</p> 6</a> I think I have about 50 5.25" floppies overall? Over the years I've accumulated approximately 30 Double Density (360kB) and 20 High Density (1.2MB) disks. Sadly, it is easier for me to write and read 5.25" disks than 3.5" disks ever since my USB floppy drive broke...</p> 7</a> As far as I'm aware, no commercial product was ever released that used the DB37 connector on the original IBM PC floppy controller ISA card. However, the pinout matches the more traditional 34-pin edge connector, so an adapter isn't difficult.</p> Porting a New Board To Migen 2017-09-25T00:00:00+00:00 Porting a New Board To Migen</h1> Taps mic</em> Is this thing on? So it's really been 11 months since I last wrote something? I really need to change that! I'm going to attempt</em> to write smaller articles in between my larger, more involved ones to keep me motivated to keep writing.</p> So today, I'm going to write about a simpler topic I meant to discuss last year, but never got around to it: How to get started with Migen</a> on your Shiny New FPGA Development Board! This post assumes you have previous experience with Python 3 and experience synthesizing digital logic on FPGAs with Verilog. However, no Migen experience is assumed!</em> Yes, you can port your a development board to Migen without having used Migen before!</p> What Is Migen- And Why Use It?</h2> Migen is a Python library (framework, maybe?) to build digital circuits either for simulation, synthesis on an FPGA, or ASIC fabrication (the latter of which is mostly untested). Migen is one</a> of</a> many</a> attempts to solve some deficiences with Verilog and VHDL, the languages most commonly used in industry to develop digital integrated circuits (ICs). If necessary, a designer can use and import Verilog/VHDL directly into their designs using one of the above languages too.</p> All the languages above either emit Verilog or an Intermediate Representation</a> that can be transformed into Verilog/VHDL. I can think of a few reasons why:</p> FPGA synthesis/ASIC front-ends are typically proprietary, can't be readily modified, and mostly only support Verilog and VHDL as input.</li> Many issues with Verilog, such as synthesis-simulation behavior mismatch, don't occur if Verilog can be emitted in a controlled manner, as is done with the above languages.^{1</sup></a> </li>} From my own experience looking at yosys, which can be used as the front- end Verilog compiler to an open-source synthesis toolchain</a>, there is little gain to targeting a file format meant for FPGA synthesis (or ASIC fabrication) directly from a higher-level language. Targeting the synthesis file format directly must be shown to be bug-free with respect to having generated Verilog and then</em> generating the synthesis file format from the Verilog; this is nontrivial.</li> </ol> The Migen manual</a> discusses its rationale for existing, but the important reasons that apply to me personally are that Migen:</p> Avoids simulation-synthesis behavior mismatches in generated output.</li> Makes it easy to automate digital logic design idioms that are tedious in Verilog alone, such as Finite-State Machines and Asynchronous FIFOs.</li> Prevents classes of bugs in Verilog code, such as ensuring each signal has an initial value and unused cases in switch</code> statements are handled.</li> Handles assignments satisfying "same sensitivity list, multiple always blocks" by concatenating them all into a single always</code> block, whereas Verilog forbids this for synthesis. In my experience, this increases code clarity.</li> </ul> As an added bonus, I can write a Migen design once and, within reason, generate a bitstream of my design without needing a separate User Constraints File (UCF) for each board that Migen supports. This facilitates design reuse by others who may not have the same board as I do, but has a new board with the required I/O peripherals anyway.</p> For the above reasons, I am far more productive writing HDL than I ever was writing Verilog.^{2</sup></a> </p>} Choice of Tools Disclaimer</h3> Of the linked languages above, I have only personally used Migen. Migen was the first Verilog alternative I personally discovered when I started using FPGAs for projects again in 2015 (after a 3 year break). When I first saw the typical code structure of a Migen project, I immediately felt at home writing my own basic designs, and could easily predict which Migen code would emit which Verilog constructs without reading the source. In fact, I ported my own Shiny New FPGA Development Board I just bought to Migen before I even tested my first Migen design!</p> Because of my extremely positive first impressions, and time spent learning Migen's more complicated features, I've had little incentive to learn a new HDL. That said, I maintain it is up to the reader to experiment and decide which HDL works best for their FPGA/ASIC projects. I only speak for my own experiences, and the point of me writing this post is to make porting a new board to Migen easier than when I learned how to do it. The code re-use aspect of Migen is important to me, and when done correctly, a port to a new board is very low-maintenance.</p> Leveraging Python To Build FPGA Applications</h2> To motivate how to port a new development board, I need to show Migen code right now. If you haven't seen Migen before now, don't panic! I'll briefly explain each section:</p> from migen import * from migen.build.platforms import icestick class Rot(Module): def __init__(self): self.clk_freq = 12000000 self.ready = Signal() self.rot = Signal(4) self.divider = Signal(max=self.clk_freq) self.d1 = Signal() self.d2 = Signal() self.d3 = Signal() self.d4 = Signal() self.d5 = Signal() ### self.comb += [j.eq(self.rot[i]) for i, j in enumerate([self.d1, self.d2, self.d3, self.d4])] self.comb += [self.d5.eq(1)] self.sync += [ If(self.ready, If(self.divider == int(self.clk_freq) - 1, self.divider.eq(0), self.rot.eq(Cat(self.rot[-1], self.rot[:-1])) ).Else( self.divider.eq(self.divider + 1) ) ).Else( self.ready.eq(1), self.rot.eq(1), self.divider.eq(0) ) ] </code></pre> If you've never seen Migen code before, and/or are unfamiliar with its layout, I'll explain some of the more interesting points here in comparison to Verilog:</p> from migen import * from migen.build.platforms import icestick </code></pre> Migen by default exports a number of primitives to get started writing Modules</code>. Other Migen constructs, such as a library of useful building blocks (migen.genlib</code>) and Verilog code generation (migen.fhdl.verilog</code>) must be imported manually. migen.build.platforms</code> contains all the FPGA development platforms Migen supports; the goal of this blog post will be to reimplement the iCEstick</a> development board for use from Migen.</p> </li> class Rot(Module): </code></pre> A Module</code> is the basic unit describing digital behavior in Migen. Connections between Module</code>s are typically made by declaring instance variables that can be shared between Module</code>s, and using submodules</code>. Submodule syntax is described in the manual, and is required to share connections between modules.</p> </li> self.ready = Signal(1) </code></pre> A Signal</code> is the basic data type in Migen. Unlike Verilog, which requires keywords to distinguish between data storage (reg</code>) and connections/nets (wire</code>), Migen can infer from a signal's usage whether or not the signal stores data. The 1</code> indicates the signal is one-bit wide.</p> </li> self.divider = Signal(max=self.clk_freq) </code></pre> In addition to providing a bit width, I can tell Migen the maximum value a Signal</code> is expected to take, and Migen will initialize its width to log2(max) + 1</code>. However, nothing prevents the signal from exceeding max when run in a simulator or FPGA.</em></p> </li> ### self.comb += [j.eq(self.rot[i]) for i, j in enumerate([self.d1, self.d2, self.d3, self.d4])] </code></pre> By convention, data-type declarations are separated from code describing how code should behave using ###</code>. Migen statements relating connections of Signal</code>s and other data types must be appended to either a comb</code> or sync</code> attribute. Connections are typically made using the eq</code> function of Signal</code>s. Python's slice notation accesses individual or subsets of bits of a Signal</code>.</p> comb</code>, or combinationial, statements are analogous to Verilog's assign</code> keyword or combinational always @()</code> blocks, depending on the Migen data type/construct. In combinational logic, the output immediately changes in response to a changing or just-initialized) input without any concept of time (in theory).</p> </li> self.sync += [ If(self.ready, If(self.divider == int(self.clk_freq) - 1, self.divider.eq(0), self.rot.eq(Cat(self.rot[-1], self.rot[:-1])) ).Else( self.divider.eq(self.divider + 1) ) ).Else( self.ready.eq(1), self.rot.eq(1), self.divider.eq(0) ) ] </code></pre> By contrast, appending to the Migen sync</code>, or synchronous, attribute emits Verilog code whose output only changes in response to a positive edge</em> clock transition of a given clock, using the following syntax: module.sync.my_clk += [...]</code>. If the clock is omitted, the clock defaults to sys</code>.</p> In synchronous/sequential logic, outputs do not change immediately in response to a changing or just- initialized input; the output only registers a new value based on its input signals in response to a low-to- high transition of another, usually periodic, signal. Migen only always @(posedge clk)</code> blocks, so a negedge</code> clock must be created by inverting an existing clock, such as self.comb += [neg_clk.eq(~pos_clk)]</code>.</p> As one might expect, If().Elif().Else()</code> is analogous to Verilog if</code>, else if</code>, and else</code> blocks. Migen will generate the correct style of always</code> block to represent signal transitions regardless of whether the If()</code> statement is part of a comb</code> or sync</code> block.</p> </li> </ul> I omitted discussing any Migen data types, input arguments to their constructors, other features provided by the package, and common code idioms that I didn't use above for the sake of keeping this blog post on point. Most of the user-facing features/constructors are documented in the user manual. I can discuss features (and behavior) not mentioned in the manual in a follow-up post.</p> Adding a New Board</h2> The above code</a> was adapted from the rot.v</a> example of Arachne PNR. In words, the above code turns on an LED, counts for 12,000,000 clock cycles, then turns off the previous LED and lights another of 4 LEDs; after 4 LEDs the cycle repeats. A fifth LED is always kept on as soon as the FPGA loads its bitstream.</p> Our goal is to get this simple Migen design to work on a new FPGA development board, walking through the process. Since this example is tailored to the iCE40 FPGA family, I'm choosing to port the iCEstick board to Migen... which already has a port...</p> Interactive Porting</h3> My original intention was to write this blog post while</em> I was creating the icestick.py</code> platform file to be committed into migen.build</code>. Unfortunately, at the time, Migen did not have any support for targeting the IceStorm synthesis toolchain.^{3</sup></a> So I ended up}implementing</a> the IceStorm backend and doing my blog post as intended went by the wayside while debugging.</p> That said, I'm going to attempt to simulate the process of adding a board from the beginning. I will only assume that the IceStorm backend to Migen exists, but the icestick.py</code> board file does not.</p> We Need a Constraints File</h3> Before I can start writing a board file for iCEstick, we need to know which FPGA pins are connected to what. For many boards, the manufacturer will provide a full User Constraints File (UCF) with this information. For demonstrative purposes^{4</sup></a> however, I will examine the schematic diagram of iCEstick instead to create my Migen board file. This can be found in a file provided by Lattice at a changing URL called "icestickusermanual.pdf".</p>} We need to know the format of FPGA pin identifiers that Arachne PNR, the place-and-route tool for IceStorm, will expect as well. The format differs for each of the FPGA manufacturers and even between FPGA families of the same manufacturer; Xilinx, for example uses [A-Z]?[0-9]</code>, as does the Lattice ECP3 family. Fortunately, IceStorm uses pin numbers that correspond to the device package, and these are easily visible on the schematic:</p> One side (port) of connections of the iCE40 FPGA to iCEstick peripherals. In this image, LED, IrDA, and one side of 600 mil breadboard-compatible breakout connections can be seen.</figcaption> </figure> If we examine the schematic and user manual, we will find the following peripherals:</p> 5 LEDs</li> 2x6 Digilient PMOD connector</li> IrDA transceiver</li> SPI Flash</li> FT2232H UART (one channel- the other is used for programming)</li> 16 Additional I/O pins</li> </ul> We might not have an actual full</em> constraints file to work with due to how arachne-pnr</code> works, but we have all the information to create a Migen board file anyway, since we have the schematics. Armed with this information, we can start creating a board file for iCEStick.</p> Anatomy of a Migen Board File</h3> Relative to the root of the migen package, migen places board definition files under the build/platforms</code> directory. All paths in this section, unless otherwise noted are relative to the package root.</em></p> Platform Class</h4> from migen.build.generic_platform import * from migen.build.lattice import LatticePlatform from migen.build.lattice.programmer import IceStormProgrammer class Platform(LatticePlatform): default_clk_name = "clk12" default_clk_period = 83.333 def __init__(self): LatticePlatform.__init__(self, "ice40-1k-tq144", _io, _connectors, toolchain="icestorm") def create_programmer(self): return IceStormProgrammer() </code></pre> A board file consists of the definition of Python class</code> conventionally named Platform</code>. Platform</code> should inherit from a class defined for each supported FPGA manufacturer. As of this writing, Migen exports AlteraPlatform</code>, XilinxPlatform</code>, and LatticePlatform</code>, and more are possible in the future. Vendor platforms are defined in a subdirectory under build</code> for each vendor, in the file platform.py</code>.</p> Each FPGA vendor in turn inherits from GenericPlatform</code>, which is defined in build/generic_platform.py</code> and exports a number of useful methods for use in Migen code (I'll introduce them as needed). The GenericPlatform</code> constructor</a> accepts the following arguments:</p> device</code>- A string indicating the FPGA device on your board.^{5</sup></a> The string format is vendor-toolchain specific; in the case of IceStorm, the format is currently "ice40-{1k,8k}-{package}"</code>.</li>} io</code>- A list of tuples of a specific format which I'll describe shortly. The list represents all non-user-expandable I/O resources on your current board. For instance, LEDs, SPI flash, and ADC connections to your FPGA would be placed in the io</code> list.</li> connectors</code>- A list of tuples with a specific layout, which I'll describe shortly. The list represents user-expandable I/O that by default is not connecte to any piece of hardware, [Pmod](https://en.wikipedia.org/wiki/ Pmod_Interface, Pmod) headers.</li> name</code>- I don't know what this input argument does exactly. Like other places in Migen with a name</code> input argument, it's meant to control how variable names are generated in the Verilog output. I don't believe it's used by any board file, so I'm ignoring it.</li> toolchain</code>- The same FPGA vendor can have multiple software suites for their FPGA families, or third-party toolchains can exist. For instance, Xilinx has both the End-of-Life ISE Toolchain and Vivado software available. Additionally, Lattice provides the Diamond toolchain for their higher-end FPGAs while Project IceStorm is an unaffialited open-source synthesis flow for the iCE40 family. Thus, the vendor platform constructors also supply</a> a toolchain</code> keyword argument to choose which toolchain to eventually invoke. In the case of iCEStick, we use Migen's icestorm</code> backend.</li> </ul> A Platform</code> class definition should also define the class variables default_clk_name</code> and default_clk_period</code>, which are used by GenericPlatform</code>. default_clk_name</code> should match the name of a resource in the io</code> list that represents a clock input to the FPGA. default_clk_period</code> is used by vendor-specific logic in Migen to create a clock constraint in nanoseconds for default_clk_name</code>. The default clock is associated with the sys</code> clock domain for sync</code> statements.</em></p> Lastly, the create_programmer</code> function should return a vendor-specific programmer. Adding a programmer is beyond the scope of this article. If a board can support more than one programming tool, the convention is to return a programmer based on a programmer</code> class variable</a> for the given board. This function can be omitted if no programmer fits, or one can be created on-the-fly using GenericPlatform.create_programmer</code>.</p> Finalization</h4> Some platforms Migen supports, such as the LX9 Microboard</a>, have a do_finalize</code> method. Finalization in Migen allows a user to defer adding logic to their design until overall resource usage is known. In particular, LX9 Microboard has an Ethernet peripheral, and the Ethernet clocks should use separate timing constraints from the rest of the design. The linked code detects whether the Ethernet peripheral was used using lookup_request("eth_clocks")</code> from GenericPlatform</code>, and adds appropriate platform constraints to the current design to be synthesized if necessary. If the Ethernet peripheral was not used in the design, the extra constraints are not added, and the ConstraintError</code> from lookup_request</code> is ignored.</p> Finalization operates on an internal Migen data structure called Fragment</code>s. Fragment</code>s require knowledge of Migen internals to use properly, so for the time being I suggest following the linked example if you need to add constraints conditionally. Of course, timing constraints and other User Constraints File data can be added at any point in your design manually using add_period_constraint</code> and add_platform_command</code> respectively, both from GenericPlatform</code>.</p> iCEStick does not have any peripherals which need special constraints, and only a single clock; Migen will automatically add a constraint for the default clock. More importantly, in the case of IceStorm/iCEStick, only a global clock constraint is supported due to limitations in specifying constraints. Therefore, I omit the do_finalize</code> method for the iCEStick board file. However, one use I have found for do_finalize</code> in platforms compatible with IceStorm is to automatically instantiate pins with pullup resistors enabled. This gets around the limitations of Arachne PNR's constraints file format without needing to instantiate Verilog primitives directly in the top level of a Migen source file, and I can show code upon request.</p> I/O and Connectors</h4> After defining a Platform</code> class for your board, all you need to do is fill in a list of _io</code> and _connectors</code> in your board file, pass them into your Platform</code>'s vendor-specific base class constructor, and Migen will take care of the rest!</p> As I stated before, io</code> and connectors</code> input arguments to the vendor-specific platform constructor are lists of tuples with a specific format. Let's start with an I/O tuple:</p> io_name, id, Pins("pin_name", "pin_name") or Subsignal(...), IOStandard("std_name"), Misc("misc")) </code></pre> An io_name</code> is the name of the peripheral, and should match the string passed into the request</code> function to gain access to the peripheral's signals from Migen. id</code> is a number to distinguish multiple copies of identically- functioning peripherals, such as LEDs. For simple peripherals, Pins</code> is a helper class which should contain strings corresponding to the vendor- specific pin identifiers where the peripheral connects to the FPGA; in the case of IceStorm, there are just the pin numbers as defined on the package pinout. I will discuss Subsignal</code>s in the next paragraph. These tuple entries are used to create inputs and output names in the Migen-generated Verilog, and provide a variable-name to FPGA pin mapping in a Migen- generated User Constraints File (UCF)</p> Without going into excess detail^{6</sup></a> , Subsignals</code> are a helper class for resources that use FPGA pins which can be seperated cleanly by purpose. The inputs to a Subsignal</code> constructor are identical to an I/O tuple entry, except with id</code> omitted. The net effect for the end user is that a resource is encapsulated as a class whose Migen Signals</code> are accessed via the class' members, i.e. comb += [my_resource.my_sig.eq(5)]</code>. This is known as a Record</code> in Migen. Records</code> also come with a number of useful}methods</a> for constructing Migen statements quickly. Think of them as analogous to C structs</code>. It is up to your judgment whether an I/O peripheral should use Subsignals</code>, but in general, I notice that Migen board files make heavy use of them.</p> The remaining inputs to an I/O tuple entry are optional. IOStandard</code> is another helper class which contains a toolchain-specific string that identifies which voltages/logic standard should use. And lastly, the Misc</code> helper class contains a space-separated string of other information that should be placed into the User Constraints File along with IOStandard</code>. Such information includes slew rate</a> and whether pullups should be enabled. These are in fact currently ignored in the IceStorm toolchain, but for my own reference I have filled them in as necessary.</em></p> A connector tuple is a bit simpler:</p> (conn_name, "pin_name, pin_name, pin_name,...") </code></pre> conn_name</code> is analogous to io_name</code>. The second element of a connector tuple is a space-separated string of pin names matching the vendor's format which indicates which pins on the FPGA are associated with that particular connector. Ideally, the pins should be listed in some order that makes sense for the connector.</p> By default, pins that are associated with connectors are not exposed by the Platform</code> via the request</code> method. Instead, a user needs to notify the platform that they wish to use the connector as extra I/O using the GenericPlatform.add_extension</code> method. Here is an example adding a PMOD I2C peripheral</a> using add_extension</code>:</p> my_i2c_device = [ ("i2c_device", 0, Subsignal("sdc", Pins("PMOD:2"), Misc("PULLUP")), Subsignal("sda", Pins("PMOD:3"), Misc("PULLUP")) ) ] plat.add_extension(my_i2c_device) plat.request("i2c_device") </code></pre> Note that adding a peripheral using add_extension</code> is similar to adding a peripheral to the io</code> list, except that the Pins()</code> element takes on the form "conn_name:index"</code>. conn_name</code> should match a tuple in the connectors</code> list, and index</code> is a zero-based index into the string of FPGA pins associated with the connector. This allows you to create peripherals on-the-fly that are (in theory) board and vendor-agnostic.</p> With the last concepts out of the way, let's jump right into creating the _io</code> and _connectors</code> list for our Platform. Each listed peripheral is implied to be a tuple inside _io = [...] or _connectors = [...])</code>:</p> ("user_led", 0, Pins("99"), IOStandard("LVCMOS33")), ("user_led", 1, Pins("98"), IOStandard("LVCMOS33")), ("user_led", 2, Pins("97"), IOStandard("LVCMOS33")), ("user_led", 3, Pins("96"), IOStandard("LVCMOS33")), ("user_led", 4, Pins("95"), IOStandard("LVCMOS33")), </code></pre> user_led</code>s are simple peripherals found on just about every development board; iCEStick has 5 of them, all identical in function (but the 5th one is green!). Resources with identical function that differ only in a pin should each be declared in their own tuple, incrementing the id</code> index.</p> Resource signal names are by convention; if a resource does not yet exist, it's up to you what you want to name the resource. However, I suggest looking at other board files for prior examples. user_led</code>, user_btn</code>, serial.rx</code>, serial.tx</code>, spiflash</code>, and audio</code> are all commonly-used I/O names used between board files.</p> ("serial", 0, Subsignal("rx", Pins("9")), Subsignal("tx", Pins("8"), Misc("PULLUP")), Subsignal("rts", Pins("7"), Misc("PULLUP")), Subsignal("cts", Pins("4"), Misc("PULLUP")), Subsignal("dtr", Pins("3"), Misc("PULLUP")), Subsignal("dsr", Pins("2"), Misc("PULLUP")), Subsignal("dcd", Pins("1"), Misc("PULLUP")), IOStandard("LVTTL"), ), </code></pre> Next we have another common peripheral- a UART/serial port. A UART peripheral makes sense to divide using Subsignal</code>s, since each pin has a distinct purpose. Although in practice most users will only use rx</code> and tx</code>^{7</sup></a> , I include all possible pins just in case. I don't remember why I included PULLUP</code> as constraints information for a majority of pins. Note that it's perfectly okay to associate a constraint with all Subsignal</code>s at once, as I do for the (unused) IOStandard</code>.</p>} ("irda", 0, Subsignal("rx", Pins("106")), Subsignal("tx", Pins("105")), Subsignal("sd", Pins("107")), IOStandard("LVCMOS33") ), </code></pre> The infrared port on iCEStick is another serial port, sans most of the control signals. I omit the optional I/O tuple/Subsignal</code> entries here, and define the IOStandard</code> similarly to the previous serial</code> peripheral.</p> ("spiflash", 0, Subsignal("cs_n", Pins("71"), IOStandard("LVCMOS33")), Subsignal("clk", Pins("70"), IOStandard("LVCMOS33")), Subsignal("mosi", Pins("67"), IOStandard("LVCMOS33")), Subsignal("miso", Pins("68"), IOStandard("LVCMOS33")) ) </code></pre> spiflash</code> and its Subsignal</code> have standardized, self-explanatory names. I suggest using these signal names when appropriate for all peripherals connected via an SPI bus</a>. I don't remember why I added the IOStandard</code> per-signal instead of all-at- once here, but the net effect would be the same either way if IceStorm made use of IOStandard</code>.</p> ("clk12", 0, Pins("21"), IOStandard("LVCMOS33")), </code></pre> Clock signals should be included as well, with at least one clock's io_name</code> matching the default_clk_name</code>. Migen may automatically request</code> a clock for your design if certain conditions are met; for now, assume you don't have to request</code> the clock.</p> ("GPIO0", "44 45 47 48 56 60 61 62"), ("GPIO1", "119 118 117 116 115 114 113 112"), ("PMOD", "78 79 80 81 87 88 90 91"), </code></pre> And lastly we have the connectors. The connector pins for GPIO0-1</code> and PMOD</code> are ordered in increasing pin order, which matches the order they are laid out on their respective connectors. This is a happy coincidence. Make sure to check your schematic and declare FPGA pins in connector order, not the other way around!</em></p> Building Our Design For Our "New" Board</h2> Now that we have created our board file, we now need to write the remaining logic to attach the board's I/O to our Rot top level</a>. Assuming the Rot</code> module is already defined, we can create a script that will synthesize our design like so:</p> if __name__ == "__main__": plat = icestick.Platform() m = Rot() m.comb += [plat.request("user_led").eq(l) for l in [m.d1, m.d2, m.d3, m.d4, m.d5]] plat.build(m, run=True, build_dir="rot", build_name="rot_migen") plat.create_programmer().flash(0, "rot/rot_migen.bin") </code></pre> Once again, I will explain each line. It should be appended to the Rot</code> top level and then run using your Python 3 interpreter:</p> plat = icestick.Platform() m = Rot() </code></pre> We create our platform and our Rot</code> module here. plat</code> contains a number of useful helper methods that help customize what is eventually sent to the synthesis flow.</p> </li> m.comb += [plat.request("user_led").eq(l) for l in [m.d1, m.d2, m.d3, m.d4, m.d5]] </code></pre> This list comprehension is how we actually connect our I/O to the LED rotation module. plat.request("user_led")</code> will create a Migen Signal</code> or Record</code> which can be used to assign to Signals</code> and Records</code> in an existing Module</code>. GenericPlatform</code> does bookkeeping to figure out which Signals</code> should be considered I/Os to/from the generated Verilog top level (and consequently, mapped to a User Constraints File).</p> Repeatedly calling request</code> for a given resource name will return a new Signal</code> or Record</code> of the same type, throwing a ConstraintError</code> exception if there are no more available resources</p> </li> plat.build(m, run=True, build_dir="rot", build_name="rot_migen") </code></pre> The GenericPlatform.build</code> function will emit Verilog output, a User Constraints File specific to your design, a batch file or shell script to invoke the synthesis flow, and any other files required as input to the synthesis toolchain. Optionally, Migen will invoke the generated script to create your design if input argument run</code> is true. build_dir</code> is self- explanatory, and build_name</code> controls the filename of generated files.</p> I pass in my top level Rot</code> Module</code> to the build</code> function. GenericPlatform</code> has internal logic to detect which signals were declared as I/O in the board file while generating Verilog input and output arguments.</p> </li> plat.create_programmer().flash(0, "rot/rot_migen.bin") </code></pre> This line is optional, but if included, Migen will invoke the iceprog</code> FTDI MPSSE programmer for iCEStick automatically after creating a bitstream using IceStorm. The first input argument is the start address to start programming, and the second argument is the name of the output bitstream. The name can be inferred from build_name</code> and the toolchain's default extension for bitstreams.</p> </li> </ul> If all goes well, and you were following along, you should now have a blinking LED example on your iCEStick! The final iCEStick platform board file is here</a>, which you can use as a reference, and I've made the Rot</code> top level available as a gist</a>. Take a look at the output files Migen generated, including the output Verilog and User Constraints File, to get a feel of how our "shiny new" board file was used!</p> Happy Hacking!</h2> If you have read up to this point, you now have some grasp on the Migen framework, and can now start using Migen in your own designs on your own development (or even deployed) designs!</p> Porting Migen to support your design takes relatively little effort, and you will quickly make up the time spent porting. Besides automating HDL idioms that are tedious to write by hand (such as FSMs), and generating Verilog that is free of certain bug classes, Migen saves time as it automates generating input files and build commands for your synthesis toolchain, and then invoking the synthesis flow automatically. Additionally, if you write your top-level Module</code> and glue code correctly, you can have a single HDL design that runs on all platforms that Migen supports, even between vendors, with much less effort than is required to do the same in Verilog alone!^{8</sup></a> </p>} Migen is certainly a step in the right direction for the future of hacking with FPGAs, and I hope you as the reader give it a try with your Shiny New Development Board like I did, and see whether your experiences were as positive as mine.</p> Acknowledgements</h2> I'd like to thank Sébastien Bourdeauducq</a>, the primary architect and maintainer of Migen, for looking over an initial version of this post and offering feedback.</p> Footnotes</h2> 1</a> For better or worse, emitting Verilog requires someone intimately familiar with the Verilog specification. Like all good specifications, it is terse, and requires a lot of memorization. But both Yosys and Migen are by and large the work of one individual each, so it can be done.</p> 2</a> In the interest of fairness, fusesoc</a> exists now to alleviate the burdern of Verilog code-sharing. I was unaware of its existence when I started using FPGAs again. I think Migen build i ntegration would be an interesting project; in general, I find setting up a board-agnostic Migen design easier than w/ FuseSoC, but importing Verilog code as a package is still incredibly useful.</p> 3</a> I erroneously assumed that because the Xilinx backend code can use the yosys Verilog compiler that there was support for support for yosys and by extension IceStorm in the Lattice backend. Not having had any Lattice FPGAs before iCEstick (yes, I jumped on the FOSS FPGA toolchain bandwagon), I never actually bothered to check beforehand!</p> 4</a> Historically, I've found that IceStorm Verilog code samples only define the pins their designs actually use. Unlike Quartus or ISE, Arachne PNR will error out instead of warn if constraints that are defined aren't actually used. Migen doesn't have this issue because it will only generate constraints that are actually used for Arachne PNR.</p> 5</a> Each Platform</code> consists of a single FPGA and attached peripherals. I assume a board with multiple FPGAs should implement a Platform</code> for each FPGA in a single board file.</p> 6</a> Subsignals also modify</a> how signal names are generated for the remainder of the synthesis toolchain.</p> 7</a> iCEStick disappoints me in that the engineers wired nearly</em> all the connections required to use the FT2232H in FT245 queue mode, which is much faster than a serial port; most dev boards do not bother connecting anything besides serial TX and RX, and possibly RTS/CTS. But alas, there's still not enough control connections to use queue mode.</p> 8</a> Portability of code gives me joy, and HDL portability is no exception.</p> NMOS IC Reverse Engineering 2016-10-12T00:00:00+00:00 NMOS IC Reverse Engineering</h1> Recently, I was tricked</del> talked</ins> into examining the die of an integrated circuit (IC)- the YM2151, a single-chip FM synthesizer. I like these old Yamaha chips, so I don't really mind doing it, but it's definitely a long term project that I expect to take months. However, one doesn't need to RE a significant portion of an IC to understand the basics of RE. Information about a chip's operation can be gleamed by examining small units, and predicting their relation to each other.</p> Information on doing IC reverse-engineering is still kind of limited, although projects like siliconpr0n</a> and whatever Ken Shirrif</a> is working on at any given time are changing that. Since I am learning how to RE ICs, I decided to document how I decoded a small ROM in the YM2151 that I suspect</em> is being used as part of the control state machine. This small ROM demonstrates the basics of REing ICs, including:</p> Separating the various layers of an IC die.</li> Mapping ROM inputs and outputs.</li> Manually reading out the ROM contents.</li> </ul> Obtaining a Die Image</h2> Before we can examine an IC die, we have to actually digitally capture an image of the die. I will not be discussing this in detail, but getting a die image typically involves:</p> Removing the IC package with corrosive chemicals, called decap.</li> Taking a number of pictures using a digital camera with a microscope.</li> Stitching all the individual images together to produce a full map of the IC die with individual transistors visible.</li> </ul> I really don't have access to the equipment to do this even at a small scale, but luckily this work was done previously in the case of YM2151 (20x magnification). I defer to Travis Goodspeed's article in section 9 of POC\|\|GTFO #4</a>) if interested in doing decap yourself.</p> Die image of YM2151 at 20x magnification, full die. The full-size image is 617MB Source</a>.</figcaption> </figure> What is NMOS?</h2> The YM2151 uses an NMOS process</a>. The NMOS logic family uses n-type metal oxide semiconductor field-effect transistors (MOSFETs) to create digital logic gates. MOSFETs are four- terminal devices that either permit or prevent current from flowing between two of the terminals, called the drain and source, depending on the voltage of the third terminal, called the gate, relative to either the drain or source. The fourth terminal, called the body, is negligible for now.</p> MOSFET Schematic Symbols</h3> The below picture shows two different types of MOSFETS: n-type depletion and n-type enhancement mode. On each MOSFET, the center terminal attached to the left column is the gate input. The source and drain are the bottom and top terminals, respectively. Each MOSFET has the source connected to a fourth terminal, which in turn connects to an arrow pointing inward to the right column, indicating n-type MOSFETs. Put simply, the arrow direction corresponds to device polarity at a specific area within the MOSFET. The source-body connection is a side effect of the MOSFET symbol I used, and we can ignore it. A segmented right column represents enhancement mode, and a solid right column represents depletion mode.</p> A depletion and enhancement mode n-type MOSFET circuit and an equivalent circuit. The depletion mode MOSFET acts as a resistor, and the enhancement mode MOSFET as a switch. A quick rationale on the schematic symbol is given in the above paragraph. On is 5V, off 0V.</figcaption> </figure> A MOSFET is a symmetrical device, so drain and source can be labeled arbitrarily. However, according to Sedra and Smith, drain is by convention always at a higher voltage than source.</p> Quick "Theory" of MOSFETs</h3> "n-type" refers to the properties of the silicon that carries current through a MOSFET. For the purposes of this blog post, I need not go into its properties.</p> In hand-calculations, a MOSFET has at least three modes of operation</a>. I'm not interested in doing simulation in this blog post (maybe in the future I will!), but for completeness, the regions are:</p> Cutoff, no conduction from drain to source.</li> Triode, linear region, acts like a resistor.</li> Saturated, increasing gate voltage does not increase current.</li> </ul> Both enchancement mode and depletion mode MOSFETs are used in the NMOS logic family. Depletion-mode MOSFETs conduct current even when the gate and source is at the same voltage; the gate must have a lower voltage than the source for cutoff to occur. They are commonly, but not exclusively</em>, used as pullup resistors, operating in the linear region and always on.</p> On the other hand, enhancement-mode MOSFETs are used to implement logic gates by either allowing current to pass or not. They operate in cutoff and saturation. The gate must be at a higher voltage than the source to conduct current. Voltage drop tends to be negligible in the enhancement-mode MOSFETs; see ratioed logic in the previous link for more information.</p> With the above two paragraphs in mind, here is an exercise: How would you implement an NMOS NOR gate (hint: MOSFETs in parallel)? NAND gate (hint: MOSFETs in series)? Inverter (NOT) (hint: Look carefully at my MOSFET image)? Notice how gates with inverters are easiest to implement? I wonder if that's a reason active-low inputs and outputs are so common in these chips?</p> Am I</em> Dealing With NMOS?</h3> How does one detect an NMOS chip? To be honest, I was told ahead of time that the YM2151 uses an NMOS process. The year that this chip was first produced (1984-5) is also a hint. Howvever, when compared to CMOS die</a> of a similar-era Yamaha chip, I notice a few differences:</p> Only one size of via in a CMOS die, many in NMOS.</li> No obvious indication of pullup MOSFETs in CMOS, prevalent in NMOS.</li> The CMOS die is more neatly organized, compared to NMOS.</li> </ul> Right now, the above list probably isn't all that meaningful :P. I'll discuss what I mean in the next section. As you may expect, my method of analysis only works for ICs made with an NMOS process. However, this is still useful for preserving many old chips where fully emulating their behavior is desired (YM2151) or even required (security chips) to preserve hardware.</p> IC Layers</h2> I decided to digitize (or vectorize) a ROM at the top of the chip, approximately one third of the length longways.</p> Section of the YM2151 I intend to vectorize, before we begin any work.</figcaption> </figure> ICs- well, not cutting-edge ones anyway- tend to be made by applying planar layers of conducting and semiconductor material. Therefore, it tends to be safe to represent each layer of material as a 2d layer in a vector image, not worrying about layer depth or wells that would be apparent in a 3d cutaway. Each layer can thus be inferred and by examining intersections and outlines left over during decap.</p> I can tell the above is a ROM from the equal-width strips of metal (remember, metal tends to be consistently colored) and the circular "holes" distributed throughout the metal strips. These "holes" are properly referred to as "vias". Vias are drilled holes, forming a connection with any layer that intersects at their location.</p> Additionally, there exist buried contacts that directly connect (typically? always?) the poly and active layers. A metal layer can be placed on top of a buried contact without creating a connection. Buried contacts can frequently be identified by a light square outline where poly and active intersect, but this is not guaranteed. Sometimes buried contacts must be inferred from context. The takeaway here is: vias and buried contacts form two additional logical layers that need to be vectorized.</p> Buried contacts are different from MOSFET gates b/c a MOSFET gate is not a direct connection. A gate has a layer of insulating oxide separating the polysilicon and the active layer/wells underneath. However, it tends to be obvious from context and visual inspection which type of connection exists, even without having a 3d cutaway view which would show the differences.</p> Not all ICs have the same number of layers or the same layer material type, but in the case of NMOS, it's safe to divide an IC die into at least three layers:</p> Metal, conducting material. Typically a whiteish hue that stands out, in the case of aluminum (which is what YM2151 uses).</li> Polysilicon, pure silicon. Used for MOSFET gates. Color cannot be assumed, but outlines are obvious.</li> Active, doped silicon used for drain and source. Color cannot be assumed, but outlines are obvious.</li> </ul> Let's Digitize!</h2> With the above out of the way, let's digitize the metal layer and vias of the ROM. Please note that in some images that I may miss a section :P. I correct it in a later step unless otherwise noted.</p> Layer Color Table</caption> Layer</td> Color</td></tr></thead> Metal</td> Yellow</td></tr> Polysilicon</td> Red</td></tr> Active</td> Blue</td></tr> Via</td> Green</td></tr> Buried Contact</td> Pink</td></tr> </tbody> </table> Metal layer is easily visible. Here, the ROM matrix, ROM inputs and power and ground rails are digitized. I will show how to infer the latter three shortly.</figcaption> </figure> Here, we have digitized most of the vias of interest.</figcaption> </figure> The image has become a bit crowded after digitizing the metal and via layers, so for the time being I will disable them.</p> Let's start looking for transistors. I personally like to start with pullups, becausepullups have a very distinctive shape on an NMOS die, and the power rail can also be inferred. As mentioned before, NMOS pullups are depletion-mode MOSFETs, and they have very large gate widths to create a current path even without an applied gate voltage. Additionally, to provide the pullup effect, there exists a connection to the active layer on the source side of the MOSFET that looks like a hook.</p> Pullup MOSFETs thus tend to look like "rectangles with a hook", with slightly more emphasis on the hook to create the source-to-gate connection. We can now safely vectorize the pullups, and immediate polysilicon traces emanating from the pullups.</p> We have vectorized our first MOSFET gate! A depletion mode MOSFET at that.</figcaption> </figure> All pullups, polysilicon layer, vectorized. By process of elimination we know the remaining two sides of the ROM are the inputs or outputs.</figcaption> </figure> In our depletion mode pullups, the active layer consisting of source and drain runs through the center of the wide gate. We can trace out the active layer completely now, but I deliberately stopped short. The crossing of two layers near the pullups at the bottom is signficant.</p> We can start vectorizing the active layer that forms the source and drain of the pullups.</figcaption> </figure> Notice that each strip of the metal layer connecting at the top of the ROM terminates in a via. The via connects to a layer below, either active or poly, that runs across the length of the ROM. This layer abruptly terminates after crossing the active layer that directly connects to the pullup's source. There was a transistor formed due to that crossing during fabrication! We can safely assume them to be enhancement mode transistors used as switches due to the gate size.</p> Our unknown layer must</em> be poly because they form the gate of a transistor. Furthermore, Because the metal at the top of the ROM attaches directly to the gate of a transistor for each input, the top of the ROM must be our input. By process of elimination, the output of our ROM is on the right.</p> Our first enhancement mode transistors. Notice how much smaller the gate is for each compared to depletion mode.</figcaption> </figure> Now I decided to take a break from the poly and vectorize the buried contacts. The buried contacts in this section are all visible as squares at poly and active crossings. Since a pullup must</em> have a buried contact to connect the gate and source, let's start with the pullups. Can you find the outline of the other buried contacts before scrolling to the second image?</p> Some buried contacts due to pullup connections have been digitized. All buried contacts of interest in this section are visible.</figcaption> </figure> Remaining buried contacts of interest digitized.</figcaption> </figure> Now, I finish the active layer, which by process of elimination is going to be the remaining unvectorized traces. These form a number of enhancement- mode MOSFET switches distributed through the ROM matrix. Anywhere the poly crosses the active layer is a transitor!</em></p> Remaining active layer forming the ROM matrix digitized. I accidentally missed part of the active layer at the bottom of the second column when taking these images.</figcaption> </figure> With the active layer (minus my mistake) digitized, the ROM has been fully vectorized. I re-enable the metal and via layers to show the final result. Additionally, I vectorized a few more sections all all layers, only one of which is relevant to the ROM.</p> The thick metal trace below the ROM which connects to the active layer of the ROM matrix (at the source terminals of the enhancement mode MOSFETs immediately attached to depletion mode pullups), is in fact a ground trace. From experience, I can expect the active columns running through the ROM matrix to be connected to ground. I will explore why in the next section.</p> Fully vectorized ROM, plus some extra connections.</figcaption> </figure> Schematic Capture</h2> I am arbitrarily labeling the leftmost and bottommost bus lines the LSBs of the input and output, respectively. Thus, bit positions increase as one travels from left to right and bottom to top of the ROM matrix.</p> Initially, I had intermediate images of my progress creating the schematic. Unfortunately, for various formatting reasons (repeated transistor numbers, inconsistent resolution), the intermediate images didn't turn out how I liked, so I removed them.</p> I created the schematic in a manner very similar to how I vectorized starting with the pullups, then adding the inputs and their corresponding MOSFET switch connections. Then I added the ROM outputs. Next, I added the remaining wires that run down the ROM matrix columnwise, which attach to the source of the switch enhancement mode MOSFETs and pullup depletion-mode MOSFETs respectively. I finished schematic capture by adding the additional switch transistors that exist anywhere the active layer crosses poly within the matrix.</p> For each trio of column wires, the leftmost wire is the ROM input, the middle wire is the active layer running across each row of the ROM matrix, and the rightmost wire is attached to its corresponding pullup.</p> Full schematic. This 5x10 ROM contains nearly 60 transistors!</figcaption> </figure> I made a mistake when drawing the above schematic. By convention, the source should be at a lower voltage than the drain, but for the transistors within the ROM matrix, I accidentally swapped source and drain. In an IC, this does not matter, as a MOSFET is symmetric and swapping source and drain does not affect device operation (for our intents and purposes). However, without this disclaimer, I'm sure I will confuse people. Perhaps the drain and source distinction is best ignored for this schematic.</p> Reading Out the ROM Contents</h2> With the above schematic, we can gleam some interesting information about how the ROM works. I assume the ROM inputs are either always a valid 1 or 0, because I am assuming that this ROM is driven by internal control logic.</p> If any given bus input is 0, the input will not turn on the switch transistors at the bottom of the ROM, placed immediately before pullups. This means that the pullups are not actively driven low, and the source terminal of the pullups remains at a high logic value. The logical high is propogated to all transistors whose gates are connected to the pullup source; these transistors are on. All transistors whose gates are attached to the bus input are in cutoff and have no effect on circuit operation.</p> Notice that the metal corresponding to each bit output is attached to all transistors in a row in parallel. This means that if any</em> of the transistors in a given row are on, the entire metal row, and consequently the output, is pulled low as well. This is also called wired-AND</a>.</p> In a similar manner, if any given bus input is 1, the input will turn on the switch transistor and the logical level at the pullup source will be driven low. All transistors whose gates are attached to the pullup source terminal will be in cutoff and will not drive the metal strips low. However, because the bus input is logical high, any transistors whose gate is attached to the bus input will drive its corresponding bit output low.</p> We now have enough information to devise boolean expressions and a truth table for the entire ROM!</p> Output Bits Driven Low For Each Input Bus Line</caption> Bus Line+Value</td> Output Bits Driven Low</td></tr></thead> I_{0</sub> High</td>} O_{1</sub>, O_{6</sub>, O_{7</sub></td></tr>}}} I_{0</sub> Low</td>} O_{0</sub>, O_{5</sub>, O_{8</sub>, O_{9</sub></td></tr>}}}} I_{1</sub> High</td>} O_{0</sub>, O_{3</sub>, O_{6</sub>, O_{7</sub></td></tr>}}}} I_{1</sub> Low</td>} O_{1</sub>, O_{4</sub>, O_{5</sub>, O_{8</sub>, O_{9</sub></td></tr>}}}}} I_{2</sub> High</td>} O_{2</sub>, O_5</sub>} I_{2</sub> Low</td>} O_{0</sub>, O_{1</sub>, O_{3</sub>, O_{4</sub>, O_{6</sub>, O_{7</sub>, O_{8</sub>, O_{9</sub></td></tr>}}}}}}}} I_{3</sub> High</td>} O_{1</sub>, O_{2</sub>, O_{3</sub>, O_{6</sub>, O_{7</sub></td></tr>}}}}} I_{3</sub> Low</td>} O_{0</sub>, O_{4</sub>, O_{5</sub>, O_{8</sub>, O_{9</sub></td></tr>}}}}} I_{4</sub> High</td>} O_{6</sub></td></tr>} I_{4</sub> Low</td>} O_{0</sub></td></tr> </tbody> </table> Output Bus Line Equations</caption> Bus Line</td> Boolean Expression</td></tr></thead> O_0</sub></td> ~(~I_{0</sub> \| I_{1</sub> \| ~I_{2</sub> \| ~I_{3</sub> \| ~I_{4</sub>)</td></tr>}}}}} O_1</sub></td> ~(I_{0</sub> \| ~I_{1</sub> \| ~I_{2</sub> \| I_{3</sub>)</td></tr>}}}} O_2</sub></td> ~(I_{2</sub> \| I_{3</sub>)</td></tr>}} O_3</sub></td> ~(I_{1</sub> \| ~I_{2</sub> \| I_{3</sub>)</td></tr>}}} O_4</sub></td> ~(~I_{1</sub> \| ~I_{2</sub> \| ~I_{3</sub>)</td></tr>}}} O_5</sub></td> ~(~I_{0</sub> \| ~I_{1</sub> \| I_{2</sub> \| ~I_{3</sub>)</td></tr>}}}} O_6</sub></td> ~(I_{0</sub> \| I_{1</sub> \| ~I_{2</sub> \| I_{3</sub> \| I_{4</sub>)</td></tr>}}}}} O_7</sub></td> ~(I_{0</sub> \| I_{1</sub> \| ~I_{2</sub> \| I_{3</sub>)</td></tr>}}}} O_8</sub></td> ~(I_{0</sub> \| ~I_{1</sub> \| ~I_{2</sub> \| ~I_{3</sub>)</td></tr>}}}} O_9</sub></td> ~(I_{0</sub> \| ~I_{1</sub> \| ~I_{2</sub> \| ~I_{3</sub>)</td></tr> </tbody> </table> Extracted ROM Contents of Analyzed Section. Underscores are for clarity.</caption> Input Bus</td> Output Bus</td></tr></thead> 0b0_0000</td> 0b00_0000_0100</td></tr> 0b0_0001</td> 0b00_0000_0100</td></tr> 0b0_0010</td> 0b00_0000_0100</td></tr> 0b0_0011</td> 0b00_0000_0100</td></tr> 0b0_0100</td> 0b00_1100_1000</td></tr> 0b0_0101</td> 0b00_0000_1000</td></tr> 0b0_0110</td> 0b00_0000_0010</td></tr> 0b0_0111</td> 0b00_0000_0000</td></tr> 0b0_1000</td> 0b00_0000_0000</td></tr> 0b0_1001</td> 0b00_0000_0000</td></tr> 0b0_1010</td> 0b00_0000_0000</td></tr> 0b0_1011</td> 0b00_0010_0000</td></tr> 0b0_1100</td> 0b00_0000_0000</td></tr> 0b0_1101</td> 0b00_0000_0000</td></tr> 0b0_1110</td> 0b11_0001_0000</td></tr> 0b0_1111</td> 0b00_0001_0000</td></tr> 0b1_0000</td> 0b00_0000_0100</td></tr> 0b1_0001</td> 0b00_0000_0100</td></tr> 0b1_0010</td> 0b00_0000_0100</td></tr> 0b1_0011</td> 0b00_0000_0100</td></tr> 0b1_0100</td> 0b00_1000_1000</td></tr> 0b1_0101</td> 0b00_0000_1000</td></tr> 0b1_0110</td> 0b00_0000_0010</td></tr> 0b1_0111</td> 0b00_0000_0000</td></tr> 0b1_1000</td> 0b00_0000_0000</td></tr> 0b1_1001</td> 0b00_0000_0000</td></tr> 0b1_1010</td> 0b00_0000_0000</td></tr> 0b1_1011</td> 0b00_0010_0000</td></tr> 0b1_1100</td> 0b00_0000_0000</td></tr> 0b1_1101</td> 0b00_0000_0001</td></tr> 0b1_1110</td> 0b11_0001_0000</td></tr> 0b1_1111</td> 0b00_0001_0000</td></tr> </tbody> </table> As we can see, a number of inputs result in zero state outputs, and the MSB only changes the output of two ROM entries depending on whether its set or not. Perhaps a number of these states are illegal and just given a default output? I wonder what this ROM is used for? When I figure it out, I'll make an edit to this page!</p> Future Direction</h2> As readers can probably see by now, digitizing and REing old ICs is completely doable, if tedious. Personally, I would say it's more mechanical than reversing a binary with IDA or radare2, once you know what to look for. However, like REing a binary, it does take a long time to fully RE an IC.</p> There are tools to automate the schematic capture process of an IC, and aid in analysis as well. Olivier Galibert's dietools</a> are one example that I hope to discuss in future posts.</p> As of writing this post (October 12, 2016), my work in vectorizing and schematic capture of the YM2151 can be found here</a>.</em></p> Anecdote</h3> Back in 2011, I discovered that the MAME project was decapping ICs to defeat security/protection circuits on old arcade boards that prevented them from being emulated properly. The me in 2011 thought this was the most fascinating thing, the "last bastion" of proper accurate emulation. I never thought I would have the skill set required to do IC analysis.</p> Even up until summer 2016, I said that I wouldn't do IC reverse engineering, despite preservation of old technology being important to me. I felt it was beyond my comprehension, and that I would not be able to learn how to identify features in a reasonable amount of time. With help from others, I was wrong, and I'm glad that I was. If you're on the fence about learning a new technical subject, don't hesitate. We're all smart, and filled with doubt. Others will be willing to help!</p> Thanks</h3> I would like to thank members of siliconpr0n for looking over this post, especially Olivier Galibert for correcting a few mistakes. Additionally, I'd like to thank Digi-Key for their extremely useful Scheme-it</a> schematic program, which I used to create the schematics (including the nice arrow!).</p> Floppy Disk Notes 2016-02-10T00:00:00+00:00 Floppy Disk Notes</h1> What follows is a set of links that I've collected over the years that give technical information on how to encode/decode floppy disks signals, as well as the theory of operation behind this dying medium.</p> I find reading these documents important for preservation purposes, especially since computer programmers of the past relied on these engineering details for copy protection purposes.</p> Additionally, a deep understanding of internals may one day help to recover data that is thought to be lost using statistical analysis of the read signal.</p> Links are ordered relatively in the order I read them/I recommend reading them, and sections tend to build upon each other.</p> General</h2> The Floppy User Guide</a></dt> A good technical overall technical description of how a floppy drive accesses data.</dd> </dl> Floppy Drive</h2> SA800/801 Diskette Storage Drive Theory of Operations</a></dt> Without question, the most important document on this list. If you read any document, read this. It's not quite enough information to build a floppy drive from scratch, but it's enough to bring someone interested up to speed. Hard to believe this document is 40 years old in 2016!</dd> SA850/SA450 Read Channel Analysis Internal Memo</a></dt> This internal memo donated by a Shugart employee includes a floppy drive read head transfer function analysis based on experiments Shugart did in the late 70's.</dd> </dl> Phase-Locked Loops (PLLs)</h2> Phaselock Techniques, Floyd M. Gardner</a></dt> A monograph on analog PLLs. Does not discuss All-Digital PLLs (ADPLLs).</dd> NXP Phase Locked Loops Design Fundamentals Application Note</a></dt> A quick reference for analog PLL design.</dd> </dl> Encodings</h2> MFM</abbr></h3> Floppy Disk Data Separator Design Guide for the DP8473</a></dt> To be written.</dd> Encoding/Decoding Techniques Double Floppy Disc Capacity</a></dt> Gives background on more complicated physical phenomenon associated with floppy drive recording, such as magnetic domain shifting.</dd> Floppy Data Extractor</a></dt> A schematic for a minimum component data separator that does not require a PLL, but uses a digital equivalent. Perhaps a simple ADPLL</abbr>?</dd> </dl> RLL</abbr></h3> IBM's Patent for (1,8)/(2,7) RLL</a></dt> I'm not aware of any floppy formats that use (2,7) RLL, but hard drives that descend from MFM floppy drive encodings do use RLL. RLL decoding is far more involved than FM/MFM.</dd> </dl> GCR</abbr></h3> This is a format used by Apple II drives and descendants. Software has more control over this format, so there are more opportunities for elaborate data protection compared to the IBM platforms. TODO when I have time to examine non-IBM formats. Track Formats</h2> IBM 3740 (FM, Single Density)</h3> TODO. Described in Shugart's Theory of Operations manual.</p> IBM System 34 (MFM, Double Density)</h3> TODO. Described in various documents on this page, but I've not yet found a document dedicated to explaining the format.</p> Floppy Disk Controller ICs</h2> NEC 765</h3> 765 Datasheet</a></dt> The FDC used in IBM PCs. It is not capable of writing raw data at the level of the IBM track formats. Thus, attempting to write copy-protected floppies is likely to fail with this controller.</dd> 765 Application Note</a></dt> NEC created an application note to discuss how to integrate the 765 into a "new" system, either using DMA or polling on receipt of interrupts.</dd> </dl> TI TMS279X</h3></h3> TMS279X Datasheet</a></dt> Includes a diagram of the IBM System 34 track format.</dd> </dl> NI DP8473</h3> DP8473 Datasheet</a></dt> A successor to the 765 that is capable of handing formats such as 1.2MB High Density (HD) disks</dd> Design Guide for DP8473 in a PC-AT</a></dt> TODO. It appears I lost my original commentary on this document.</dd> </dl> Floppy Disk Controller Cards</h2> IBM PC FDC Card (765)</a></dt> Includes schematics. The PLL circuit on the last page is in particular worth analyzing.</dd> </dl> If anyone has any interesting new documents to add, please feel free to contact me, and I will add them to this page with credit!</p>}}}}}

WDJ

How To Download Your Twitter Archive Using `curl`

2024-09-18T00:00:00+00:00

How To Download Your Twitter Archive Using curl</code></h1> Without fail, for the past several months, every time I've tried to download my Twitter archive from my web browser, I've not been able to do so. The download gets to about 50 or so megabytes, downloading at like 50 kilobytes per second, before uncermoniously failing: I couldn't tell you why downloading from my browser fails. Maybe it's a Firefox bug. Maybe it's a Twitter bug^{1</a> . Maybe my hardware is old. It could be a combination. Regardless of the reason, I want to download my Twitter archive, and I'm being prevented from doing so.} Each time I remember to creating an up-to-date archive, I try the direct browser link, completely forgetting that it's not going to work. And each time, I have to relearn the process I went through to successfully download my archive. So this is my attempt to document it for future-me and others. All subsequent text assumes that you have requested your Twitter archive, and that Twitter has notified you that your archive is ready. I suggest trying to download no later than 4 days after Twitter notified you, to give you a few days to experiment before Twitter deletes the archive. What Doesn't Work- A Direct Link And wget</code> Or curl</code></h2> After watching the download fail a couple of times ^{2</a> , I copied the download link to my clipboard and pasted it into wget</code> and curl</code> in frustration:} I knew in my gut that this would not work and it cannot possibly work. Yet I did it anyway. I am not the only one</a> either. The simple solutions are tempting when one is frustrated. The reason the download fails is that Twitter wants some extra data in the HTTP headers</a>^{3</a> .}wget</code></a> and curl</code></a> do not- and cannot- provide these headers without your help. You can use your web browser to find the correct data needed. What Does Work- Your Web Browser Monitor And curl</code></h2> Preparing The Browser And Monitor</h3> Both Firefox and Chromium-based browsers provide tools to get the required data- the Network Monitor. This post focuses on Firefox because that's what I use, but the process for Chromium should be similar. To prepare Firefox and Monitor, I open a new Firefox window, and set the address to about:blank</code>^{4</a> . The Monitor can be accessed by right-clicking the about:blank</code> page, and selecting Inspect</code> from the pop-up:} Clicking Inspect</code> opens the Firefox Developer Tools. The Network Monitor tab in Developer Tools should look something like below: By default, Developer Tools opens up to the Inspector tab. You need to switch over to the Network tab to open the Network Monitor.</figcaption> </figure> Collecting Headers</h3> The web browser and Monitor are prepared. We will get the data we need to make a curl</code> suceed at downloading our Twitter archive succeed by first using our newly-opened browser window to (start) download(ing) our Twitter archive. You can get the direct download link to your archive by right-clicking the "Download archive" from the "Donwload an archive of your data" page, like so: Then, copy the link to the fresh about:blank</code> page browser window we just opened: Once you press enter in the address bar of that about:blank</code> page, the Network Monitor will collect data about HTTP(S) request(s) the browser is making behind the scenes. As a side effect of the HTTP(S) request(s), your browser should open a Save Dialog. You can ignore it. The important headers that curl</code> needs should be a row in the Network Monitor tab. As of this writing (2024-09-18), accessing the direct archive download link from the about:blank</code> page should result in a lone request captured by the Monitor. The request should be a file of the form twitter-{{date}}-{{hash}}.zip</code>; that's your archive! Right clicking this request gives you several options to "fill in" the missing required data to download your archive using curl</code> or other tools. You may want to try all the options to experiment.^{5</a></figcaption> </figure>} If you right-click this row, you will get several options to copy the headers missing from your initial</a> wget</code> or curl</code> command. In my case, I have chosen Copy as cURL (POSIX)</code>, which copies a curl</code> command-line invocation with the relevant arguments/header to your clipboard. After I paste the clipboard curl</code> command into my terminal with an explicit output file, the download begins at a reasonable speed. Most importantly, the download doesn't fail halfway through^{6</a> :} I provide an extra --output</code> argument to curl</code> to specify where to write the archive data, e.g. curl https://example.com/example.zip --output example.zip </pre> Without --output</code>, curl</code> will refuse to download your archive, because ZIP files</a> are binary data. The --output</code> argument is not part of the clipboard data from Monitor!</figcaption> </figure> The net effect of going to developer console and creating a curl</code> invocation is that Twitter cannot tell the difference between your web browser and curl</code> downloading your archive. All is well in my archiving world for now^{7</a> .} Failure Modes</h3> Sometimes curl</code> may fail to download the archive, even with the relevant headers provided by the browser. Despite the lack of error output, this curl</code> invocation has failed to download an archive. Me forgetting the output</code> parameter is unrelated to the error above (although that is indeed another error :)).</figcaption> </figure> My experience is that the time window that Twitter allows you to download the link is short. If you see the above image, you need to redo the verification to download your archive and get back to the download link page, like here: Getting back to this page should re-verify you, and should refresh the time window that the download link is valid. However, don't click the download link this time! The curl</code> invocation you copied from the browser inspector earlier should still be valid, even after you get back to the direct download link page. Instead, run curl</code> again with the params added by Monitor and an --output</code> param, and the download should succeed^{8</a> .} Happy Downloading! Make Copies!</h2> It's not clear to me how long Twitter will last. For various reasons, I want to have an archive of my data in case the site goes under, especially considering that present-me and future-me often need to be reminded of the context of past-me^{9</a> .} I back up my archive irregularly; maybe I should do something about that. But when I do attempt a backup, I am persistent. I don't know if it'll be the last backup I can make before my backups are the only access to (that portion of) past-me that I have left. Even if the web browser direct download works for you, let this post be a reminder to make an archive of your data if you so wish</a>. In the meantime, I'll work on following my own advice. Footnotes</h2> 1</a> I feel like there's been a lot more of these Twitter bugs since late 2022. Couldn't tell you why, it's a mystery... 2</a> I also got locked out of accessing the download link at all due to repeated failures for 24 hours. Go me. 3</a> At the very least, Twitter wants cookie</a> headers. But other headers may be required, and I didn't experiment to figure out the mandatory ones. 4</a> It's not clear to me when Monitor starts collecting data after the Developer Tools are open. To make looking at Monitor's data easier on my eyes, I prefer starting from a blank page. This step may not be required, but I prefer to not deviate from what I know works. 5</a> In my case, although I run Firefox on Windows, I ultimately wanted to store my Twitter archive on an NAS</a> running Linux. Using curl</code> on Linux, I can skip the intermediate download to a Windows machine. I'm sure curl</code> on PowerShell</a> works fine too. 6</a> Even if it did fail, I vaguely recall being able to use curl</code> to resume the download from where I left off. 7</a> All is never well in my archiving world. I'll be organizing data until I'm six feet under, I'm afraid... 8</a> At least, I hope it succeeds. If you try the procedure in this post and it fails, I'm interested to know. Contact me</a> in all the usual places! 9</a> In fact, it's not clear to me that the method described will continue to work for future-me (today is 2024-09-18). For instance, Google seems to have introduced elaborate code to stop</a> youtube-dl</code> from working, and that already has to emulate</a> a decent chunk of a browser due to Javascript. I'll have to deal with that when the time comes.

Using `z3` To Solve Logic Puzzles 2023-08-15T00:00:00+00:00 Using z3</code> To Solve Logic Puzzles</h1> Recently, I learned about point-free programming</a>. The day after I learned about it, I was idly wishing</a> for a puzzle book on converting program code to point-free form. A point-free programming puzzle book would be a good exercise on how well I can compose</a> functions. Sadly, I'm not aware of such a puzzle book, nor am I qualified to write one. However, thinking of puzzles again reminded me of how I liked logic puzzle</a> books when I was younger (anyone remember PennyPress</a>?). A few years ago, I solved a logic puzzle using an SMT solver out of curiosity. I was motivated to do this after seeing Claire Wolf</a> doing the same</a> to solve a similar "guess the combination to the lock given hints" puzzle. I'd intended to write a blog post about how I mapped the word problem of a logic puzzle to an SMT query, but I never followed up on it. I know my last post was in late 2019. 2020 through 2023 has not been kind to the world and our collective social/mental bandwidth. Mine included. So while the thought of fun logic puzzles from my youth were fresh in my head yesterday (2023-06-30), I forced myself remember how I solved a logic puzzle with an SMT solver 5 years ago. And at last I'm going to document it here. Prerequisites</h2> This post assumes a background in Boolean logic</a> and/or one programming language (along with how to use its Boolean operators). Familiarity with S-expressions</a> is also a plus, because the input to the z3</code></a> program we're using uses S-expressions</a>. However, I will be explaining all the input we send to z3</code> line-by-line. So feel free to learn about S-expressions at your own pace while or after reading this post. When referencing concepts common in other programming languages, I refer to Rust</a> documentation, because at this writing (2023-07-01), it's easiest for me to find the analogous concepts in the Rust language's documentation. And lastly, you should know what a logic puzzle</a> is. Enjoying solving logic puzzles is a nice bonus :). What is an SMT Solver?</h2> Before I explain what an SMT solver is, and how to use them to solve puzzles, I need to explain what SAT and SAT solvers are as prerequisites. SAT</h3> Suppose you have some Boolean expressions- a bunch of Boolean variables chained together- with Boolean operators- &&</code>, ||</code>, !</code>, etc. Here is an example query: A || (B && !C) B && C </code></pre> SAT, or SATisfiability, is the process of determining whether a set of Boolean expressions like above can all be made true</code> (1</code>) for one or more combination of the values for the given Boolean variables. Given n</code> Boolean variables, where each variable has 2 possible values- true</code> (1</code>) and false</code> (0</code>)- you can determine whether there's an assignment that satisfies all the Boolean expressions by "substituting all combinations of values for the variables in the expressions" (brute-forcing). For instance, here's how we can brute-force the above query in (up to) 8 (2^3</code>) easy steps: A=0,B=0,C=0</code>: UNSAT</code> 0 || (0 && !0) = 0 0 && 0 = 0 </code></pre> </li> A=1,B=0,C=0</code>: UNSAT</code> 1 || (0 && !0) = 1 0 && 0 = 0 </code></pre> </li> A=0,B=1,C=0</code>: UNSAT</code> 0 || (1 && !0) = 1 1 && 0 = 0 </code></pre> </li> A=1,B=1,C=0</code>: UNSAT</code> 1 || (1 && !0) = 1 1 && 0 = 0 </code></pre> </li> A=0,B=0,C=1</code>: UNSAT</code> 0 || (0 && !1) = 0 0 && 0 = 0 </code></pre> </li> A=1,B=0,C=1</code>: UNSAT</code> 1 || (0 && !1) = 1 0 && 1 = 0 </code></pre> </li> A=0,B=1,C=1</code>: UNSAT</code> 0 || (1 && !1) = 0 1 && 1 = 1 </code></pre> </li> A=1,B=1,C=1</code>: SAT</code> 1 || (1 && !1) = 1 1 && 1 = 1 </code></pre> </li> </ol> Indeed, our query is SAT</code>isfiable when all three Boolean variables are set to 1</code>. SAT Solvers</h3> When checking for satisfiability, if just one combination of variable values makes all the expressions in your query equal to 1</code>, you're done- the query is SAT</code>. Unfortunately, this means you have to try 2^n</code> combinations of values for n</code> Boolean variables if you have a query that is UNSAT</code>isfiable. And even if the query is SAT</code>, you may have to try many combinations of variable values before you find a combination that returns SAT</code>, such as the above example. To make matters worse, real-world SAT problems can have thousands of expressions and millions of variables. Unlike my toy example above, it is completely impractical to try to brute-force 2^1000000</code> different combinations of Boolean variables. It is believed</a> that for some SAT problems, there is no better strategy than brute forcing all 2^n</code> combinations. However, real-world SAT problems are better behaved and can be solved efficiently using, for instance, the Davis–Putnam–Logemann–Loveland</a> (DPLL) algorithm or Conflict-Driven Clause Learning</a> (CDCL) algorithm. SAT solvers are a type of computer program optimized for solving Boolean satisfiability problems. They take advantage of these algorithms and hueristics common to real-world SAT problems to efficiently find whether a query is SAT</code> or UNSAT</code>^{1</a> .} SMT And SMT Solvers</h3> Satisfiability Modulo Theory (SMT) problems and solvers are what you get when you're no longer limited to queries of just Boolean variables. Queries can be taken over Booleans, reals, integers, arrays of Booleans, functions, enumerations, and many more! You can even create queries of combinations of these data types! As might be expected, by relying on queries over data types besides Booleans can result in dramatic speedups, as well as dramatic increases in complexity :). While some data types map nicely^{2</a> to Boolean expressions such as}bitvectors</a>, modern SMT solver use theories to map more complex data types back to Boolean expressions. Just like how Booleans are governed by the theory of Boolean algebra, each data type an SMT solver supports has a theory that explains "what you can and cannot do" with the data type. For instance, consider the following expressions, comparing equality of integers I</code>, J</code>, and K</code>: (I == 1) && (J == I + 1) (I == 2) || (K == J + 3) (I == 2) && (J == K + 2) </code></pre> The expressions I == 1</code>, J == I + 1</code>, I == 2</code>, K == J + 3</code>, J ==K + 2</code> and all yield Boolean values, and can therefore be converted to and from Boolean values A</code>, B</code>, C</code>, D</code>, and E</code> respectively: A && B C || D C && E </code></pre> While Boolean expressions can be solved for using the above algorithms for SAT, solving for equality is governed by the congruence closure</a>, and the theory of integers is governed by, well, number theory</a>. The DPLL/CDCL algorithms can be modified to handle conversion to and from Boolean expressions, as well as handle congruence closure and other theories. This is known as the DPLL(T) framework^{3</a> .} z3</code> And SMT-LIB</h3> Our SMT solver of choice will be Microsoft Research's z3</code></a>. Other SMT solvers exist, such as yices</code></a>, and thanks to each using different algorithms, there is not a single SMT solver that solves all practical SMT problems better than the others. Speedups and slowdowns can be dramatic depending on which SMT solver you use^{4</a> . However, a query to solve a logic puzzle will not tax any SMT solver, so seeing that I'm familiar with its}API</a>, I chose z3</code>. For our purposes, z3</code> is invoked as the following: z3 -smt2 model.txt </code></pre> model.txt</code> is a text file we will create in SMT-LIB</a> format. SMT-LIB is an S-expression</a>-based text input/output format widely-supported by SMT solvers, z3</code> included. It contains commands to construct input queries for many different theories, run the solver on your query via check-sat</code>, and examine the solver's output. In particular, after running check-sat</code>, you can use SMT-LIB to: Display the value of the variables the solver used to satisfy your query (get-model</code>, where the values are written in terms of define-fun</code>).</li> Display a proof that your query is UNSAT</code> (get-proof</code>).</li> Evaluate</a> functions that the solver knows about, including functions defined for you</a> by the solver (get-value</code>)!</li> </ul> As is custom for an S-expression format, SMT-LIB has a large amount of parantheses to represent nested</a> data structures. While I show a generic example of how to use all our needed SMT-LIB commands, I will be skimming over the meaning of the parentheses nesting for this example application. An Example In SMT-LIB Format</h4> Like SAT solvers, SMT solvers can handle Boolean expressions just fine, as long as they're in the SMT-LIB format. Using the SAT example</a> earlier in the post, the equivalent SMT-LIB query would look something like this: (set-option :produce-models true) (set-logic QF_UF) (declare-const a Bool) (declare-const b Bool) (declare-const c Bool) (assert (or a (and b (not c)))) ; A || (B && C) (assert (and b c)) ; B && C (check-sat) (get-model) </code></pre> There are a few things to note comparing the SAT example I created above: The first line set-option :produce-models true</code> enables the get-model</code> command. The second line set-logic</code> restricts the data types and theories that the solver will use. The SMT-LIB website provides</a> a catalog of logics that can be used with set-logic</code>, and QF_UF</code> is a simple sub-logic suitable for a Boolean-only query. Although z3</code> tolerates the first two lines missing, its good practice to add those lines for other solvers like yices</code> and for optimizing. Assuming that some data types and theories can't occur in a given query sometimes allows the solver to speed up proofs. </li> </ol> Variables are not implicit in SMT-LIB, and need to be declared with declare-const</code>. declare-const</code> takes a variable name, and a type- Bool</code> in this case, but an SMT solver understands other types. During check-sat</code>, the solver will fill in a value for you for each variable. The value of a variable is the same everywhere that the variable is used, hence the const</code> part. </li> As is traditional with S-expressions, operators such as and</code>, or</code>, etc are written in prefix notation</a> rather than infix notation</a>. Parentheses are required around operators and their arguments to actually evaluate</a> the expression/do the calculation^{5</a> . </li>} The Boolean expressions you want to check for satisfiability aren't implicit either; you must tell the solver about your query using one more more assert</code>s. The solver must find a way to make all assert</code>s true</code> simultaneously for your query to return SAT</code>. A query without any assert</code>s returns true</code></a>. </li> Comments are preceded with ;</code>. </li> check-sat</code> checks whether your query is SAT</code> or UNSAT</code>, and nothing else. In contrast, get-model</code> returns the value of the variables the solver used to satisfy your query if your query was SAT</code>. In this case, the output of the solver, including the variable values chosen, matches the SAT example: sat ( (define-fun b () Bool true) (define-fun a () Bool true) (define-fun c () Bool true) ) </code></pre> Additionally, there is also get-value</code>. As will become clear later, get-value</code> is sometimes more useful than get-model</code> for presenting information about a model which satisfies your query. get-value</code> is especially when you're interested in the return values of uninterpreted functions</a>. </li> Although not represented in the above example, there are a few additional operators available to the Bool</code> theory in SMT-LIB that are not available in a SAT solver due to being polymorphic</a>: =</code>, distinct</code>, and ite</code>. =</code> checks whether two generic types T</code> have equal value. If T</code> is Bool</code>, this is equivalent to XNOR</code>. Unlike many programming languages, SMT-LIB uses single-equals (=</code>), not double-equals (==</code>) for comparison. </li> distinct</code> checks whether two generic types T</code> are not of equal value. If T</code> is Bool</code>, this is equivalent to XOR</code>. </li> ite</code> corresponds to if</code>-then</code>-else</code> expressions in your favorite programming language. The if</code> predicate requires Bool</code>, while the then</code> and else</code> branches of the expression require a T</code> as input. The entire expression returns a T</code> as output. It returns the value of the then</code> branch when the if</code> predicate evaluates to true</code>, and returns the value of the else</code> branch when the if</code> predicate evaluates to false</code>. When T</code> is Bool</code>, ite</code> has the following truth table: I</th> T</th> E</th> O</th></tr></thead> 0</td> 0</td> 0</td> 0</td></tr> 0</td> 0</td> 1</td> 1</td></tr> 0</td> 1</td> 0</td> 0</td></tr> 0</td> 1</td> 1</td> 1</td></tr> 1</td> 0</td> 0</td> 0</td></tr> 1</td> 0</td> 1</td> 0</td></tr> 1</td> 1</td> 0</td> 1</td></tr> 1</td> 1</td> 1</td> 1</td></tr> </tbody></table> </li> </ul> </li> </ol> While the above example doesn't really use an SMT solver's true power, it does illustrate the fundamentals of how to interface with one. We will discuss how to use other datatypes and theories with SMT-LIB as we turn a logic puzzle into an SMT query. SMT And Logic Puzzles</h2> With the whirlwind introduction to SMT out of the way, my goal with this post is to show you how to map the word problem presented by a logic puzzle to a SAT</code>isfiable SMT query. I will introduce an example logic puzzle first, and then show how to use various SMT-LIB commands to create some datatypes and constraints to represent the puzzle. At a high level, if you visualize a logic puzzle grid, such as the one below, the conversion process looks like this: The SMT theory of algebraic datatypes will be used to represent the headers of each row and column of data.</li> Correlating each row to the correct columns of data will be represented by the theory of uninterpreted functions .</li> Clues/hints and constraints implicit in a logic puzzle will be represented by assert</code>s.</li> </ul> This empty logic puzzle contains 9 rows and columns, along with headers for each row and column. The column headers, in left-to-right order, are: Batman, Jake, Dibii, Lazer, Sleep, Ball, 1, 2, and 3. The row headers in top-to-bottom order, are: Ruby, Spot, and Starbuck, 1, 2, 3, Lazer, Sleep, and Ball. Each set of 3 row and column headers belong to the same category: Batman, Jake, Dibii are tomcats.</li> Ruby, Spot, and Starbuck are queen</a> cats.</li> Lazer, sleep, and ball are activities.</li> 1, 2, and 3 are number of kittens.</li> </ul> To solve the puzzle, you are intended to mark each table entry a ✓ if the row and column are associated with each other, or ✗ if the row and column are not associated with each other, based on clues given to you (not represented in the above image). A table entry whose row and column are of the same category does not need to be marked. The pictured table represents this by completely omitting such entries, making the table similar to an upper triangular matrix</a>. The bottom part of the puzzle consists of a table with the columns queen, tom, activity, and number of kittens. The column queen is filled in for you with Ruby, Spot, and Starbuck. You fill the remaining rows of this table based on which row and columns you marked with a ✓ in the above table. I got the logic puzzle template (before filling in the headers) from Daydream Puzzles</a>.</figcaption> </figure> An Example Puzzle</h2> For this post, I'll be using a puzzle from Aha! Puzzles</a> as an example. I'm not going to copy the puzzle description, since a person presumably spent time to create the puzzle (among others), and I don't want to steal their work. Instead I will paraphrase the puzzle and grid, which describes a set of cats. Description</h3> There are three queens</a> named Ruby, Spot, and Starbuck.</li> Each queen has a partner, favorite activity, and number of kittens. As is customary for this type of problem, it is implied that each Queen has a different partner, favorite activity, and number of kittens.</li> The possible partners (toms, as in tomcats) are: Batman, Jake, and Dibii.</li> Activities include: chasing a Lazer, Sleeping, and chasing a Ball.</li> Each queen/tom pair has between 1 to 3 kittens (inclusive).</li> </ul> Given the above information, and the following clues, figure out each queen's partner, favorite activity, and the number of kittens they have. Clues</h3> The puzzle is intended to have a unique answer. However, using only the description above, there are multiple correct answers. The clues below are intended to limit the multiple correct answers down to a single unique answer^{6</a> :} Batman's partner likes to chase a ball. She was not Starbuck.</li> Dibii's partner liked the lazer.</li> Ruby was a cat who liked to sleep.</li> Starbuck has two more kittens than whoever Batman's companion is.</li> </ol> Mapping The Puzzle To An SMT Query</h2> Now that we have a puzzle, we can map the textual description to a query and a mathematical model that the SMT solver will return. Let's create the query piece by piece, starting with initial setup code: (set-option :produce-models true) (set-logic ALL) </code></pre> The first two lines we've already seen, but we have a new logic ALL</code>. set-logic ALL</code> is a bit of a cop-out; SMT-LIB allows ALL</code> for applications which generate queries on-the-fly. ALL</code> is not intended for handwritten queries. Unfortunately, z3</code> doesn't</a> really</a> have that many logics that support datatypes. And none of the logics that do support datatypes seem to support both integers and uninterpreted functions.^{7</a> So, without any better options for now, I'm just making explicit that z3</code> should use "whatever means necessary" to solve the logic puzzle}^{8</a> .} Enums</h3> declare-datatypes</code> is part of the most recent version of SMT-LIB, version 2.6. It is a very general function that allows you to create complex data types, including recursive</a> data types. In our usage we will only be using it to create the equivalent of enum</code></a>s without data in other programming languages^{9</a> .} In general, you create an enum</code> Foo</code> whose variants A</code>, B</code>, C</code> (and more, if wish) have data using the following syntax^{10</a> :} (declare-datatypes ((Foo 0)) (((A) (B) (C)))) </code></pre> Anywhere in your query where a value of type Foo</code> is used, the SMT solver has no choice but to make the value either A</code>, B</code>, or C</code>. Additionally, the following properties hold^{11</a> :} A = A</code>, B = B</code>, and C = C</code> (reflexivity</a>)</li> A != B</code>, B != C</code>, and A != C</code> (antisymmetry</a>).</li> </ul> This can be confirmed with the following query, which returns SAT</code>: (declare-datatypes ((Foo 0)) (((A) (B) (C)))) (assert (and (= A A) (= B B) (= C C))) (assert (and (distinct A B) (distinct A C) (distinct B C))) (check-sat) </code></pre> With the above syntax and properties in mind, the names of the cats and their activities correspond very nicely to enum</code>s whose variants don't have data: (declare-datatypes ((Tom 0)) (((Batman) (Jake) (Dibii)))) (declare-datatypes ((Queen 0)) (((Ruby) (Spot) (Starbuck)))) (declare-datatypes ((Activities 0)) (((Lazer) (Sleep) (Ball)))) </code></pre> Uninterpreted Functions</h3> Functions form the backbone of programming languages and mathematics, and SMT formulas are no exception. We can create pure functions</a> to abstract away common parts of our query using define-fun</code>. Here is an example add</code> function that "adds one to its input Int</code>": (define-fun add ((x Int)) Int (+ x 1)) (check-sat) (get-value ((add 1) (add 2) (add 10))) </code></pre> The above query returns SAT</code></a> in response to check-sat</code>. After check-sat</code>, your model has an add</code> function that the solver can call. We don't have any assert</code>s in the above query, so get-model</code> returns nothing. However, we can display the result of calls to our add</code> function using get-value</code>. get-value</code> takes a single argument, but this single argument can take an arbitrary number of terms. Each of the three terms to the get-value</code> call above will make the solver read, evaluate, and print</a> the results of calling your add</code> function on various integers- exciting! sat (((add 1) 2) ((add 2) 3) ((add 10) 11)) </code></pre> Not only can we define functions, we can also declare functions, without bothering to define/fill in the function body! We can request the solver to fill in the function body for us by using declare-fun</code>: (declare-fun add_two (Int) Int) (check-sat) (get-model) (get-value ((add_two 1))) </code></pre> The above query returns sat</code> in response to check-sat</code>. get-model</code> will then return the definition of add_two</code> that the solver created: sat ( (define-fun add_two ((x!0 Int)) Int 0) ) (((add_two 1) 0)) </code></pre> In this case add_two</code> returns the literal 0</code>. Not particularly interesting. What if we add an assert</code>, to force add_two</code> to return 3</code> when given the input 1</code>? (declare-fun add_two (Int) Int) (assert (= (add_two 1) 3)) (check-sat) (get-model) (get-value ((add_two 1))) </code></pre> sat ( (define-fun add_two ((x!0 Int)) Int 3) ) (((add_two 1) 3)) </code></pre> It's a function that returns 3</code> for all inputs. Now the solver is being cute. "Technically correct" is the best type of correct. The solver won't create a more interesting function body unless forced to via proper constraints on the input arguments and return values^{12</a> .} We will constrain our uninterpreted functions shortly, but for now, let's declare 4 uninterpreted functions with declare-fun</code>. These functions will in fact return the answers for our logic puzzle, once the solver fills them in: (declare-fun QueenPartner (Queen) Tom) (declare-fun QueenActivity (Queen) Activities) (declare-fun QueenKittens (Queen) Int) (declare-fun TomPartner (Tom) Queen) </code></pre> To be specific: QueenPartner</code> returns the Tom</code> partner of the given Queen</code>.</li> QueenActivity</code> returns the given queen's favorite activity.</li> QueenKittens</code> returns the number of kittens a given Queen</code> has as an Int</code>.</li> TomPartner</code> returns the Queen</code> partner of the given Tom</code>.</li> </ul> Implicit Constraints</h3> By their very nature, logic puzzles contain some usually-implied rules that we must follow when solving them if we don't want incoherent, nonsensical answers: Each possible choice in each category is different from all the other choices. It should be apparent that, for instance, Batman and Dibii are different tomcats (uniqueness).</li> There are a finite number of choices for each category, and this number is the same for each category. E.g. each of the three queens can choose from one of three activities. There is no fourth "secret" activity that the logic puzzle knows about and we don't (finiteness).</li> As mentioned earlier</a>, each queen has a mutually-exclusive partner, number of kittens, and activity with respect to other queens.</li> We can also deduce that a tom has the same number of kittens and activity as its queen partner.</li> </ul> A logic puzzle grid such as above</a> visually makes it easy to follow these rules. Unfortunately, an SMT solver knows nothing about logic puzzles without our help, so we must explain those implicit constraints in our query. We use assert</code>s to force an SMT solver to reject nonsensical-but-valid answers to a query of variables, where in the case of a logic puzzle, our variables are our uninterpreted functions. Our enum</code>s we created via declare-datatype</code> already encode these uniqueness/finiteness constraints mentioned above. We can also represent the number of kittens each cat pair has as a NumKitten</code>s enum</code>, but for convenience that will become clear shortly, I chose to use an Int</code>. While Int</code>s have a uniqueness constraint (1</code> is different than 2</code>), in SMT-LIB, Int</code>s do not have a finiteness constraint. Thus without additional information, the solver free to set NumKitten</code>s for each cat pair to 1</code>, 10</code>, or even 1,234,567</code>, among other fine Int</code>s, if the solver so chooses. The puzzle's description implies</del> outright says that the number of kittens each cat pair must be between 1</code> and 3</code> inclusive. We can constrain the values for number of kittens the solver will try using an assert</code>: ; Integer constraint (assert (and (>= (QueenKittens Ruby) 1) (<= (QueenKittens Ruby) 3) (>= (QueenKittens Spot) 1) (<= (QueenKittens Spot) 3) (>= (QueenKittens Starbuck) 1) (<= (QueenKittens Starbuck) 3))) </code></pre> While the uniquess constraints of enum</code>s and Int</code>s gets us far, it's not quite enough. As quoted above</a>: As is customary for this type of problem, it is implied that each queen has a different partner, favorite activity, and number of kittens. </blockquote> Although the solver recognizes that each tom is a different cat, without additional information, nothing prevents the solver from giving an answer which assigns, for instance, two queens to the same tomcat partner. We can further constrain the solver from trying to assign two queens the same partner, number of kittens, and activity by constraining the uninterpreted functions</a> we defined earlier via more assert</code>s. For instance, the statement (assert (not (= (QueenPartner Ruby) (QueenPartner Spot)))</code> tells the SMT solver "However you decide to fill the function body, I need the function QueenPartner</code> to not return the same value for Ruby</code> and Spot</code>". To properly constrain our query, you need one assert</code> for each possible pair of values in each category, which can be combined into a single large conjuction</a>^{13</a> :} ; "Only one dot in each row/col" consistency check. (assert (and (not (= (QueenPartner Ruby) (QueenPartner Spot))) (not (= (QueenPartner Ruby) (QueenPartner Starbuck))) (not (= (QueenPartner Spot) (QueenPartner Starbuck))) (not (= (QueenKittens Ruby) (QueenKittens Spot))) (not (= (QueenKittens Ruby) (QueenKittens Starbuck))) (not (= (QueenKittens Spot) (QueenKittens Starbuck))) (not (= (QueenActivity Ruby) (QueenActivity Spot))) (not (= (QueenActivity Ruby) (QueenActivity Starbuck))) (not (= (QueenActivity Spot) (QueenActivity Starbuck))))) </code></pre> The above check is morally equivalent to a human making sure a logic grid has only one ✓ per row and column for each combination</a> of values in each category. Because of the uniqueness constraints for each enum</code> and Int</code>, the above is sufficient to make sure the solver knows that each queen has a mutually exclusive partner, number of kittens, and activity and that such a solution exists^{14</a> .} Lastly, each cat pair shares a favorite activity and the number kittens they have. So far, we've only written constraints for each queen in the cat pair. However, we don't need to rewrite the constraints for each tom! There is another consistency check we need, and a side-effect, we can represent each tom's favorite activity and number of kittens for free in terms of this constraint. The TomPartner</code> uninterpreted function returns the Queen</code> who is the partner of the given Tom</code>. Likewise, the QueenPartner</code> function returns the Tom</code> whose partner is the given Queen</code>. Without extra constraints, the SMT solver doesn't know "obvious" facts like "'the partner of the partner' of a cat is the same cat you started with"^{15</a> . We can represent this fact with another assert</code>:} ; All other constraints for Tom are implied if the partner of the partner is the same. (assert (and (= (QueenPartner (TomPartner Batman)) Batman) (= (QueenPartner (TomPartner Jake)) Jake) (= (QueenPartner (TomPartner Dibii)) Dibii))) </code></pre> After this constraint, TomPartner</code> unambiguously maps</a> a tom back to queen. Since a cat pair has the same favorite activity and number of kittens, the solver now has enough information to solve for each tom's favorite activity and number of kittens in terms of queens. Explicit Constraints</h3> The implicit constraits are necessary but not sufficient</a> to constrain the logic puzzle to a unique answer. If an SMT solver is presented with only the above lines, but there will be multiple ways to fill in our uninterpreted functions which satisfy that query. The solver will choose to present to you^{16</a> one interpretation for each function out of all valid ones, and chances are, it won't be the intended answer to the logic puzzle based on the clues.} If you incorporate the clues, however, the SMT solver should^{17</a> be limited to only one valid answer to your query. In this sense, the clues given}above</a> form the explicit, outright-stated constraints unique to a specific logic puzzle. Explicit constraints are handled identically to the implicit constraints in the previous section</a>, by mapping a textual description of the clues described earlier to asserts</code>: Batman's partner likes to chase a ball. She was not Starbuck. </blockquote> ; 1. Batman chose the female who liked to chase a ball, but she was not Starbuck. (assert (= (QueenActivity (TomPartner Batman)) Ball)) (assert (not (= (TomPartner Batman) Starbuck))) </code></pre> Dibii's partner liked the lazer. </blockquote> ; 2. Dibii's companion liked to chase the laser light. (assert (= (QueenActivity (TomPartner Dibii)) Lazer)) </code></pre> Ruby was a cat who liked to sleep. </blockquote> ; 3. Ruby loved to cuddle up to her male for a long afternoon nap in the sun. (assert (= (QueenActivity Ruby) Sleep)) </code></pre> Starbuck has two more kittens than whoever Batman's companion is. </blockquote> This clue is the reason I represented the number of kittens as an Int</code>. While I would have to reinvent addition if I used a NumKittens</code> enum</code>, the SMT solver already knows the theory of Int</code>s and can add them using +</code>^{18</a> .} ; 4. Starbuck had two more kittens than Batman's companion. (assert (= (QueenKittens Starbuck) (+ (QueenKittens (TomPartner Batman)) 2))) </code></pre> Getting An Answer</h3> Once, we add check-sat</code>, our query text file is finished, and we can feed it to z3</code> using the invocation described</a> to check whether our query is satisfiable. You can download the completed query here</a> and run it online</a> (as of 2023-08-13) if you wish. Many times, knowing SAT</code> or UNSAT</code> is "good enough". But for a logic puzzle, we actually want the variable values. get-model</code> works, but this shows the definition of the uninterpreted functions you asked the solver to fill in. While seeing the function bodies are useful information, for solving a logic puzzle, it's probably better to use get-value</code> to see individual outputs of each now-interpreted function for "interesting" inputs. To compare and contrast, I use all of check-sat</code>, get-model</code>, and get-value</code>: (check-sat) (get-model) (echo " Spot:") (get-value ((QueenPartner Spot) (QueenKittens Spot) (QueenActivity Spot))) (echo " Ruby:") (get-value ((QueenPartner Ruby) (QueenKittens Ruby) (QueenActivity Ruby))) (echo " Starbuck:") (get-value ((QueenPartner Starbuck) (QueenKittens Starbuck) (QueenActivity Starbuck))) </code></pre> Running the entire query through my local copy of z3</code> gives the following output: sat ( (define-fun QueenKittens ((x!0 Queen)) Int (ite (= x!0 Spot) 1 (ite (= x!0 Starbuck) 3 2))) (define-fun QueenPartner ((x!0 Queen)) Tom (ite (= x!0 Spot) Batman (ite (= x!0 Starbuck) Dibii Jake))) (define-fun QueenActivity ((x!0 Queen)) Activities (ite (= x!0 Starbuck) Lazer (ite (= x!0 Ruby) Sleep Ball))) (define-fun TomPartner ((x!0 Tom)) Queen (ite (= x!0 Jake) Ruby (ite (= x!0 Dibii) Starbuck Spot))) ) Spot: (((QueenPartner Spot) Batman) ((QueenKittens Spot) 1) ((QueenActivity Spot) Ball)) Ruby: (((QueenPartner Ruby) Jake) ((QueenKittens Ruby) 2) ((QueenActivity Ruby) Sleep)) Starbuck: (((QueenPartner Starbuck) Dibii) ((QueenKittens Starbuck) 3) ((QueenActivity Starbuck) Lazer)) </code></pre> First off, the query is sat</code>, which is a good start! The model returned by get-model</code> isn't hard for me to follow either. I could fairly quickly figure out each answer to the logic puzzle using get-model</code> alone, and I'm sure anyone else reading this post could too. But using get-value</code> to evaluate the functions the solver filled in for us gives us the logic puzzle answers directly, ready to be written down! So let's save ourselves the trouble and populate the puzzle's answers thanks to get-value</code>! Was the solver correct? Well, as of this writing (2023-07-23), Aha! Puzzles lets you solve logic puzzles interactively online. I'll just let the below screenshot speak for itself: The popup says "Congratulations, you win!", so indeed z3</code> was correct. Although the logic puzzle grid and answer table is obscured by the popup, there is just enough of the grid and answer table shown to verify that I filled in the puzzle with the answers returned by z3</code>.</figcaption> </figure> Conclusion/Next Steps</h2> I still find logic puzzles fun to work through manually. To me, asking a machine to solve every logic puzzle for you defeats the satisfaction of working your brain to solve the puzzle yourself. So while I hope the above example provides some insight as to how to map word problems to an SMT query/model, I wouldn't suggest that you start solving every logic puzzle with an SMT solver. That said, I found that trying to map a logic puzzle to an SMT solver query was also intellectually stimulating. It is a low-stakes problem that gave me the opportunity to exercise my skills at creating SMT queries, as opposed to my typical usage where I have a program generate</a> queries and models for me. I was excited after I got my query working, and I wanted to document the result. Speaking of generating queries and models, it's possible to take this logic puzzle-solving use case further by creating a Domain-Specific Language</a> (DSL). Like how the previously-linked SymbiYosys converts Verilog to SMT-LIB queries, this DSL would convert a more convenient textual representation of a logic puzzle into SMT-LIB queries. It's a bit more up-front work than writing SMT-LIB queries manually, and probably not worth the time spent other than for learning. However, since this project is meant to be low-stakes and for fun, I may look into a DSL in the future for a follow-up post. It's about time I wrote a(nother</a>) compiler anyway. Thanks</h2> I would like to thank my friends Screwtape</a> and Alyssa Rosenzweig</a> for their general feedback and a few last-minute corrections on the final draft of this post. Footnotes</h2> 1</a> I will not be discussing how to write a SAT solver; for those interested I suggest this tutorial</a> for building a SAT solver in Python. The tutorial is based on Donald Knuth's</a> "SAT0W" solver. Although I have not personally done so yet, I would probably also look at Knuth's "SAT10" solver, which is based on the Davis–Putnam–Logemann–Loveland</a> (DPLL) algorithm. 2</a> To be honest, I'm going by what Wikipedia says</a> here. There's no external reference: Early attempts for solving SMT instances involved translating them to Boolean SAT instances (e.g., a 32-bit integer variable would be encoded by 32 single-bit variables with appropriate weights and word-level operations such as 'plus' would be replaced by lower-level logic operations on the bits) and passing this formula to a Boolean SAT solver. </blockquote> 3</a> Just like for SAT solvers, I will not be discussing how to write an SMT solver. While in the process of writing this post, I have been learning about the DPLL(T) framework myself. I've found the following set of slides useful: Lecture 7</a> of Emina Torlak's</a> CSE507</a>. In particular, Emina's slides outline the DPLL(T) algorithm in terms of Conflict-Driven Clause Learning</a> (CDCL), an extension of DPLL.</li> Lecture 2</a> of Wes Weimer's</a> CS590</a>. </ul> 4</a> In my own experience, yices</code> is significantly faster at many problems involving formal verification of HDL. However, it is not universal. This is why the Symbiyosys</a> front end for HDL verification will run the same query against multiple SMT solvers in parallel if you have them installed. 5</a> If you don't include parentheses to actually evaluate operators, weird things you probably didn't intend will happen. For instance, try comparing the result of the query (assert (= or true false true))</code> to (assert (= (or true false) true))</code>. 6</a> More succinctly, the clues "constrain the solution space". 7</a> The logic that's closest to what I need, Quantifier-Free Finite Domain</a> (QF_FD</code>), has some additional restrictions</a> which cause my logic puzzle query to return UNKNOWN</code>. I'm not actually certain why it returns UNKNOWN</code>, but I suspect I'm running afoul of the restrictions by using datatypes with uninterpreted functions, and not just equality (=</code>) comparisons. 8</a> I belive `ALL` is z3</code>'s default logic anyway. 9</a> declare-datatypes</code> is not required, and the logic puzzle can be solved without them. In fact, I have constructed a working query without datatypes</a> for this logic puzzle- twice</a>! I wanted to solve the puzzle with yices</code> as well as z3 as an exercise; while yices</code> does support enum</code>s and the like, they are not compatible with SMT-LIB. As shown in the linked yices</code> examples, declare-sort</code>, declare-const</code>, and declare-fun</code>, plus some assert</code> constraints can emulate declare-datatypes</code> and finite integers reasonably. However, for a handwritten example, I wanted to focus on creating constraints appropriate to a logic puzzle rather than reinventing enum</code>s. 10</a> The integer argument to declare-datatypes</code> is used for parametric polymorphism</a>. It represents, informally, "the number of types to be filled in later". 11</a> What A</code>, B</code>, and C</code> actually are in terms of "bits in memory somewhere" doesn't matter as long as the solver manipulates these values of type Foo</code> according to the above rules. 12</a> To make the example more interesting, I tried to use assert</code>s to get the solver to generate a function body which uses +</code> (integer addition), but got bored and gave up. This post is getting long anyway. 13</a> Readers familiar with using z3</code> may notice that I can more-succinctly represent the exclusivity constraints using a forall</code> quantifier</a>. Indeed, this works, and was how I originally</a> solved the example logic puzzle. Except for an extra pair</code> datatype</a>, the quantifier version of our query is very similar to the quantifier-free query</a> constructed in this blog post; just compare the comments. However, I decided not not to discuss quantifiers in this post for a few reasons: The quantifiers are not required for enumerations with finite variants to create the constraints.</li> I am not exceptionally familiar with using quantifiers in SMT queries and when they are actually required. I haven't needed them for HDL verification, for instance.</li> Quantifiers complicate</a> the SMT solver's job. While queries without quantifiers can always be determined to be SAT</code> or UNSAT</code>, some queries with quanitifers are undecidable</a>. As of this writing (2023-07-02) I don't think I'm qualified to discuss undecidability other than in passing. </li> Not all SMT solvers support quantifiers (e.g. yices</code>).</li> </ol> Therefore, while the quantifiers are likely harmless here, I'd rather them when they are actually required, not just as a means to make queries take fewer lines. 14</a> If two enum</code> variants could be equal each other in value (i.e. not distinct</code>), it would be impossible for our uninterpreted functions to return three distinct</code> enum</code> variants to three different values of input enum</code> variants. To show this, play with this commented query online</a>: (declare-sort Foo 0) ; declare-sort declares a new type, like `Int`, `Bool`. ; It has fewer assumptions baked-in than `declare-datatypes`. ; Create some named values for our types. ; These are like our `enum` variants. (declare-const foo_one Foo) (declare-const foo_two Foo) (declare-const foo_three Foo) (declare-sort Bar 0) (declare-const bar_one Bar) (declare-const bar_two Bar) (declare-const bar_three Bar) ; Our uninterpreted function. Takes a `Foo`, returns a `Bar`. (declare-fun my-fun (Foo) Bar) ; Ensure each constant has a unique value. This is implied ; `declare-datatypes`. ; ; Change _any_ of the 6 asserts from `distinct` to `=`. ; The solver will return `unsat`. (assert (distinct foo_one foo_two)) (assert (distinct foo_one foo_three)) (assert (distinct foo_two foo_three)) (assert (distinct bar_one bar_two)) (assert (distinct bar_one bar_three)) (assert (distinct bar_two bar_three)) ; Prevent the solver from trying to create more values for our types ; from thin air. This is implied in `declare-datatypes`. (assert (and (or (= (my-fun foo_one) bar_one) (= (my-fun foo_one) bar_two) (= (my-fun foo_one) bar_three)) (or (= (my-fun foo_two) bar_one) (= (my-fun foo_two) bar_two) (= (my-fun foo_two) bar_three)) (or (= (my-fun foo_three) bar_one) (= (my-fun foo_three) bar_two) (= (my-fun foo_three) bar_three)))) ; Each input to my-fun should return a unique output. ; This is impossible unless foo_{one, two, three} have ; different values according to `=`. (assert (distinct (my-fun foo_one) (my-fun foo_two))) (assert (distinct (my-fun foo_one) (my-fun foo_three))) (assert (distinct (my-fun foo_two) (my-fun foo_three))) (check-sat) (get-model) </pre> 15</a> While I praise the SMT solver for being open-minded, polyamory is not within the scope of this logic puzzle. 16</a> By default an SMT solver will give you one answer. It is possible to get all possible answers to an SMT query. Since a logic puzzle is intended to have a single answer, ensuring that the number of unique answers is 1 would be another good consistency check at the cost of execution time. Unfortunately, I don't remember how to do this. 17</a> The puzzle should have a unique answer if it was designed properly. 18</a> I chose to write the assert</code> in terms of Int</code> and +</code> as an example. When numbers you can choose from in a category aren't sequential, and/or the clues are more complicated, I think it's useful to be able to model clues with other theories the SMT solver provides. In this case, because we know that NumKittens</code> is constrained from 1</code> to 3</code> inclusive, you could just as easily derive the constraint as: (assert (= (QueenKittens Starbuck) 3)) (assert (= (QueenKittens (TomPartner Batman)) 1)) </pre> A Tour Of ethflop 2019-12-11T00:00:00+00:00 A Tour Of ethflop</h1> ethflop</a> is a new (2019!) Terminate and Stay Resident</a> (TSR) program for IBM PCs and compatibles running a DOS flavor. ethflop will convert either your drive A:\ or B:\ floppy drive into a networked drive that uses Ethernet frames to talk to a Linux server. The Linux server runs a daemon called ethflopd</code> that can supply floppy images in place of physical disks. I first found out about ethflop from a retweet</a> of the FreeDOS Project's account. I knew then and there I at least had to try this out. It's like I was the target audience! And if you: Have old hardware "optimized" for running DOS.</li> Enjoy (responsibly or otherwise!) networking your old computers to other old and new computers.</li> Have a large number of floppy images that you downloaded over the years.</li> </ul> then I think ethflop is worth looking at! I imagine a number of vintage-oriented people who would be happy with having access to a bunch of floppy images from their DOS machine without needing to write them to physical floppies! So I've decided to take a look myself and write down my initial experiences using ethflop. My Hardware Setup (For Testing)</h2> ethflop requires two machines to use: a Linux server which runs a daemon (ethflopd</code>) that has access to some floppy images, and a machine running DOS which interfaces to the Linux daemon to access said floppy images (using the TSR portion of ETHFLOP.COM</code>). I present the hardware setup I used in the next two sections for showing off my PC AT</del> giving the rest of this post context</ins>. Feel free to skip</a> this section and refer back to it when necessary; it is mostly inessential. My 286 (DOS) Setup</h3> My 286 is an early IBM PC AT running at 6 MHz. It has 2 floppy drives- drive A:\ is a 1.2MB High Density drive, and drive B:\ is a 360kB Double Density drive. Additionally, my AT has one full height hard drive at drive C:\ (the original CMI</a> hard drive), and one half-height hard drive at D:\ that sits rather snugly in the slot under the 2 floppy drives. The C:\ drive is the boot drive and the D:\ drive is used mainly for receiving temporary files that I subsequently forget to clean up :). Picture of my IBM PC AT, showing both floppy drives. The screen includes some DOS commands I ran which are relevant to this blog post, including unloading the DOSED</code></a> autocompleter (as it interferes with CTTY</code>), loading a packet driver, and then changing the console to a serial port via CTTY</code>.</figcaption> </figure> Because of the way ethflop works, you lose access to one of the floppy drives; I chose to use drive B:\ with ethflop for testing purposes. The hard drives are unaffected, and I've yet to test whether you can reliably use ethflop with a boot floppy. If you do not have a physical drive B:\, you must use drive A:\ with ethflop. This is because DOS will emulate a virtual B:\ drive using drive A:\ if no physical drive is detected, and no B:\ drive requests will ever be sent to ethflop. My Network Interface Card (NIC) is a 3Com 3c509, which is still a fairly common ISA</a> NIC that can be even be found for sale off Ebay. 3Com still had packet drivers on their website for the 3c509 when I was searching for them in late 2010, but they pulled an Intel and they are long gone. Fortunately, the Internet Archive comes through</a> yet again! Due to me misunderstanding the Ebay listing at the time, I bought a model of 3c509 which only has thinnet (10BASE2)</a> and thicknet (10BASE5)</a> connections. Fortunately, 10BASE2 to 10BASE-T converters are still common- if way too expensive- and occassionally one can buy an old hub off Ebay with the same hardware^{1</a> as these converters. I'm not in a position to really offer concrete advice on setting up 10BASE2 right now. The moral here is get an ISA NIC with an RJ-45 connector, which is the}twisted pair (10BASE-T)</a> Ethernet that you know and love. Linux Setup</h3> I used my headless Tinker Board</a> to run the Linux daemon. My distro is DietPi, though I imagine the distro really doesn't matter; the source to ethflopd</code> is contained in a single C file and I had no troubles compiling and running it. Notably, I'm only using the Tinker Board's wifi, and I can confirm ethflopd</code> will work if used with a wlan</code> interface. I'm assuming my router takes care of converting from raw Ethernet frames to 802.11 frames, because they are not compatible</a>. This image shows a screen capture of an SSH session with the Tinker Board. The listing shows I am in the floppy image directory and I have a few floppy images prepared to share with the PC AT. I couldn't think of anything better to show off a running headless system :).</figcaption> </figure> For ease of testing, I am interfacing to DOS using a serial port and minicom</code> on the Tinker Board. It is possible to redirect DOS input and output to a serial port (COM2</code> in my case) using the CTTY</code> command. Using minicom</code> makes it much easier to capture DOS command output compared to taking photos. The picture of my AT above was taken just after I invoked CTTY</code>. Compiling and Installing</h2> Before using ethflop, one has to compile and install it first. I didn't have any issues with compiling and installing, but I've documented my experiences on how to do it for others just in case. Compiling and Installing ethflopd</code></h3> To start, download the source distribution from the ethflop website</a> using wget</code> or other means. I recommend downloading the source to a Linux machine. Once you unzip (unzip -d ethflop ethflop-20191003.zip</code>) the source distribution, you will see a directory like this: wjones@DietPi:~/src/ethflop$ ls -l total 120 -rw-r--r-- 1 wjones wjones 41865 Oct 3 16:48 ethflop.asm -rw-r--r-- 1 wjones wjones 3696 Oct 3 17:04 ethflop.com -rwxr-xr-x 1 wjones wjones 18568 Nov 25 22:15 ethflopd -rw-r--r-- 1 wjones wjones 38305 Oct 3 16:50 ethflopd.c -rw-r--r-- 1 wjones wjones 6192 Oct 3 16:55 ethflop.txt -rw-r--r-- 1 wjones wjones 803 Oct 3 17:02 Makefile </code></pre> The ETHFLOP.COM</code> tsr is written in nasm</code></a>-style assembly. While nasm</code> is provided by many Linux package repos, the author provided an already-assembled .COM</code> file for convenience. You will need to transfer this file to your DOS machine. The ethflopd</code> portion of ethflop needs to be compiled manually. Type make</code> and you will get a binary at ethflopd</code>. Any recent Linux should be able to run ethflopd</code>. As of this writing (12-8-2019), I have not tested the daemon on non-Linux systems. There's no install target, so you need to copy ethflopd</code> to its final location manually. In my case, I copied the file to my private bin directory: cp ethflopd ~/.local/bin</code>. Lastly, create a storage directory for your floppy images: mkdir ~/src/img</code>. This is the directory ethflopd</code> will use to present virtual floppy disks to the DOS machine. Optionally, you can fill this directory with your floppy images now :). Optional Access Control</h4> I recommend setting the setuid</a> bit as well so you can execute the binary without needing sudo. Since this is likely a private binary for your use alone, on a multiuser system (where you have sudo</code> privileges) you probably dont want others running ethflopd</code> either. I believe the following commands accomplish both: Change the owner and group of ethflopd</code> to root</code> and your user's primary group</a> respectively. As far as I know, the owner must be root so ethflopd</code> can create its runtime files and bind to an interface after setuid</code> takes effect. On the other hand, the group is your primary group, so you can still run the executable. sudo chown root:wjones ~/.local/bin/ethflopd </code></pre> </li> Set the setuid</code> bit on ethflopd</code>. Make sure the group owner has executable permissions while the "others" group has the executable bit cleared. This ensures that only you or root can run ethflopd</code> if that's what you want. sudo chmod 4754 ~/.local/bin/ethflopd </code></pre> </li> </ol> Installing ETHFLOP.COM</code></h3> At a minimum, you will need to transfer ETHFLOP.COM</code> and a packet driver for your particular NIC to your retro machine. If you can't find a packet driver for your card easily, tweet</a> or email</a> me and I'll see if I can help. I don't have a large collection of packet drivers myself (CNET, Kingston, 3Com), but I can point you in the right direction.^{2</a> } Once you have ETHFLOP.COM</code> and a packet driver on your retro machine, you're good to go! There is no special installation required; ETHFLOP.COM</code> is self-contained. It's probably a good idea to make sure ETHFLOP.COM</code> and your packet driver can be seen by the DOS PATH</code> though. Of course, one has to actually get ETHFLOP.COM</code> from the distribution you unzipped on your Linux machine to the vintage machine in order to actually use ethflop to its full potential. About that... Transferring ETHFLOP.COM</code> To The Target Machine</h4> Unfortunately, getting files from a modern machine to a retro DOS machine is quite a topic in and of itself. There's various ways to do this- serial port, floppy, over the network, cdrom attached to the parallel port, zip disks, etc. Depending on how much extra hardware you have, transferring a file from a new to old machine ranges from fun to very tedious. My Preferred Transfer Method- TCP/IP!</h5> It was fun for me personally to transfer ETHFLOP.COM</code> to an old machine because I had set up TCP/IP on my my PC AT years ago using mTCP</a>. mTCP applications integrate the TCP/IP stack into their binaries. The main distribution provides an FTP client (FTP</code>) and HTTP downloader (HTGET</code>), among other useful programs. I have also written my own</a> software using mTCP. For transferring files from a new to old machine, I normally prefer using the FTP</code> client, but recently, I've had trouble setting up a temporary working FTP server using pyftpdlib</code></a>. Specifically, the server claims to transfer the file, but mTCP FTP</code> terminates the transfer immediately. Instead I decided to use python3</code>'s http.server</code> module and then download the file to my AT using HTGET</code>. Specifically: On the Linux/modern side, invoke an HTTP server in the ethflop directory: wjones@DietPi:~/src/ethflop$ python3 -m http.server Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ... </code></pre> </li> On the DOS/retro side, connect to the newly-invoked HTTP server and download ETHFLOP.COM</code> using HTGET</code>: HTGET -o ETHFLOP.COM 192.168.1.162:8000/ethflop.com </code></pre> </li> </ol> In less than a second, I had ETHFLOP.COM</code> safely on my AT! The power of TCP/IP and Ethernet still amazes me. I don't remember the exact details of how I got mTCP onto my AT initially, but as I recall I had to use a third computer with a zip drive and 5.25" floppy drive as an intermediate. I used a USB Zip drive to copy files from my laptop. The files were read by the intermediate computer's internal Zip drive, and then copied onto the intermediate's 5.25" 1.2MB drive, which I then could write onto the AT's hard disk. Bootstrapping</h5> I apologize in advance for the length of this section. This needs to be spun off into a blog post by itself. Feel free to skip</a> to the next section. In the absence of a working network connection, I understand using floppy disks for data transfer to old systems is sometimes not an option. While a floppy drive is pretty much the lowest common denominator of hardware for IBM PCs, clones, and descendants, I've seen a few issues: Getting blank 5.25" floppies is inconvenient in 2019.</li> Even after you get a 5.25" drive for a second machine, I don't think any of the modern USB 5.25" floppy drive interfaces support writing disks (no demand?). So you need to setup up an intermediate machine as I described in the previous section.</li> Occassionally, I've seen people with broken floppy drives who want to install software onto their machine.</li> </ul> I think serial port is the method for transferring data between new and old machines that makes the least amount of assumptions about available hardware. A serial port in DB-9 or DB-25</a> form factor is pretty a common hardware addition to even the oldest IBM PC and clones. Serial ports are traditionally accessed through the DOS command line by the special names COM[1-4]</code> or AUX</code> (for COMmmunications or AUXilary). Modern OSes have good support for DB-9 serial ports via USB adapters</a> or PCI Express</a> because RS-232 is still widely used for better or worse. You will need a null-modem</a> cable or adapter to connect your modern and retro computer together; these are still widely available to purchase. There are a few bootstrap</a> mechanisms that have been developed over the years for DOS. They mostly involve typing in scripts (like BASIC) and/or small exectuables whose opcodes</a> or encoding</a> only use ASCII characters. Because the bootstraps only create files with ASCII characters, one can use the DOS COPY</code>command to read bootstrap programs from the serial port, such as COPY COM1 XFER.COM</code>. The ASCII restriction is because DOS uses the EOF</code> ASCII character (^Z</code>) to know when the transfer is finished.^{3</a> } Bootstrap programs often implement a minimal Kermit protocol</a> receiver program. Unlike the DOS COPY</code> command, the Kermit protocol can send and receive files with arbitrary bytes, and the protocol was designed to be extremely portable to new machines. The Kermit Project took the bootstrap problem of "transferring the file-transfer program to a new computer" seriously; all the bootstrap links I provided in this section are part of the Kermit archive at Columbia. I use either minicom</code> or Tera Term</a> as my serial transfer programs on machines with modern OSes. Both of these programs understand the Kermit protocol and can be used to send a file down the serial line as-is for the DOS COPY</code> command. However, you must manually send the EOF</code> (^Z</code>) control character yourself afterwards in both programs. I can vouch personally that msbrcv.bas</code></a> has worked for me in the past with QBASIC</code>. I have not tested in older DOS versions using BASIC</code> or BASICA</code> as of this writing (12-8-2017). I plan to test transfers with TCOM.COM</code></a> in the near future, which may be preferable since it doesn't require BASIC. Really, I should probably write a blog post on bootstrapping and move this whole section there :). Usage</h2> Starting and Stopping ethflopd</code></h3> To start the daemon, supply the following command line arguments in order: The network interface you wish to listen on (eth0</code>, wlan0</code>- wireless interfaces work, etc.) .</li> The storage directory of your floppy images.</li> </ol> My invocation looks like: ethflopd wlan0 ~/src/img/</code>. There is an optional -f</code> option to run the daemon in the foreground as well. I find this useful if you wish to invoke ethflopd</code> for as long as you're running your DOS machine, as opposed to running ethflopd</code> continuously. Once ethflopd</code> is running, it will listen for the MAC address</a> of a networked DOS computer running ETHFLOP.COM</code> to send and receive virtual floppy images. From the DOS side, detecting the MAC address of a machine running ethflopd</code> is automatic. To reiterate, there is no IP address setup because ethflop uses raw Ethernet/Wifi frames to transfer data. To stop the daemon, you need to kill</code> the PID associated with ethflopd</code> or run pkill ethflopd</code>. sudo</code> may be required depending on your particular setup. If ethflop</code> is running in the foreground, ^C</code> works fine as well :); ethflopd</code> has the same cleanup code for both kill</code> and ^C</code>. A Guided Tour Of ETHFLOP.COM</code></h3> Once the daemon is running on the Linux machine, the first thing to do on the DOS end is... read the help text :). I have copied the output of running ethflop without any arguments for your convenience! C:\>ethflop ethflop v0.6 - a floppy drive emulator over Ethernet Copyright (C) 2019 Mateusz Viste === USAGE ==================================================================== ethflop a installs the ethflop TSR as A: ethflop b installs ethflop as B: (works only if you have a real B:) ethflop i DISKNAME 'inserts' the virtual floppy named 'DISKNAME' ethflop ip DSKNAME same as 'i', but the inserted floppy is WRITE PROTECTED ethflop r OLD NEW renames virt. floppy 'OLD' to 'NEW' ethflop e 'ejects' currently loaded virtual floppy ethflop nSZ DSKNAME creates a new virt. floppy DSKNAME, SZ KB big. SZ can be: 360, 720, 1200, 1440, 2880, 4800, 8100, 9600, 15500, 31000 ethflop l displays the list of available virt. floppies ethflop d DISKNAME DELETES virt. floppy named DISKNAME - BE CAREFUL! ethflop s displays current status of the ethflop TSR ethflop u unloads the ethflop TSR === NOTES ==================================================================== * Disk names must be 1 to 8 characters long. Only A-Z, 0-9 and '_-' allowed. * ethflop requires the presence of an ethflop server on the local network. === LICENSE ================================================================== ethflop is published under the terms of the ISC license. See ETHFLOP.TXT. </code></pre> ethflop can basically be divided into three command types: Load/unload TSR (a</code>, b</code>, u</code>)</li> Query commands (l</code>, s</code>)</li> Operate on virtual disks (i</code>, ip</code>, r</code>, e</code>, n</code>, d</code>)</li> </ul> Loading/Unloading the TSR</h4> Packet Driver Installation</h5> The first</del> second</ins> thing to do is to make sure your packet driver is loaded into memory. In my case with the 3c509, this looks like: C:\>3c5x9pd 0x60 3Com EtherLink 10 ISA Packet Driver v1.2 (C) Copyright 1993 3Com Corp. All rights reserved. Error: The interrupt requested is already in use by another packet driver. </code></pre> The lone argument to 3C5X9PD.COM</code> is the x86 interrupt number to use to call into the packet driver. Interrupt 0x60</code> is a good traditional default, as indicated by the error message telling me I already loaded the driver. Some packet drivers require more options to set up, but chances are the default options provided in your NIC manual or by the supplied configuration program- as is the case with the 3c509- will work correctly. Older machines may require a bit more fine tuning, but between three different NIC manufactuters over a dozen DOS machines, I've had the default options not work exactly once^{4</a> . Manually assigning resources becomes more important if you have multiple NICs installed in a retro machine, but this is rare.} TSR Installation</h5> After you're sure a packet driver is loaded, then it's time to load the TSR! As I said before, if you only have a single physical floppy drive, you must use ethflop a</code>. However, seeing as I do have a physical B:\ drive, I installed ethflop at drive B:\: C:\>ethflop b server found at F0:03:8C:90:B2:A1 current virt. floppy: <NONE> ethflop has been installed </code></pre> ETHFLOP.COM</code> will attempt to automatically find</a> a server running ethflopd</code> without user invervention. It can take multiple tries to load the driver depending on your network connection quality. However, I've found that once ETHFLOP.COM</code> finds the server, any subsequent network errors are transient and the TSR keeps working. Once ethflop is installed, you lose access to the underlying physical drive. This also means that you can't use ethflop if you don't have a physical floppy drive.^{5</a> The virtual floppy drive presented by ethflop will be empty; you must either load or create a virtual disk before accessing the drive, or else you will get DOS' famous error message:} C:\>B: Seek error reading drive B Abort, Retry, Fail?f Current drive is no longer valid>C: C:\> </code></pre> I have not tested whether ethflop can coexist with applications using a DOS TCP/IP stack, though I can't see any immediate problems. Unloading ETHFLOP.COM</code></h5> When you're done playing with virtual floppies and/or need to access the physical drive, type ethflop u</code> to uninstall the TSR: C:\>ethflop u ethflop has been uninstalled </code></pre> You will get access back to your underlying physical floppy drive as well as the NIC after unloading. Querying commands</h4> You can get information about the ethflop TSR and what disks are available by running ethflop s</code> and ethflop l</code> respectively: C:\>ethflop s ethflop is currently installed as drive B: server is at F0:03:8C:90:B2:A1 current virt. floppy: <NONE> C:\>ethflop l msdos213 mydisk </code></pre> Uh oh, one of my virtual disks is missing from the list! As it turns out, the file extension for your images matter! I've been sloppy about this in the past, switching between .ima</code> or .img</code> on a whim. Renaming mtcpget.ima</code> to mtcpget.img</code> in my image directory on the Tinker Board makes all three images appear during a query: C:\>ethflop l msdos213 mtcpget mydisk </code></pre> Much better! Using Virtual Disks</h4> Commands to operate on virtual disks are the bulk of ETHFLOP.COM</code>'s functionality. Any typical action you would perform on a physical floppy disk has an equivalent coded into ETHFLOP.COM</code>: i</code>- Insert a virtual disk (place a disk into the disk drive).</li> e</code>- Eject a virtual disk (remove the disk from the disk drive).</li> ip</code>- Write-protect and insert a virtual disk (cover a notch or slide a switch on the disk, then insert).</li> r</code>- Rename a virtual disk (change the label on the sleeve/cover).</li> n</code>- Create a new virtual disk (go to the store and buy some floppies).</li> d</code>- Delete a virutal disk (erase a good floppy or discard a bad floppy).</li> </ul> I present these commands in the order I tried them out in my initial ETHFLOP session, which matches the list above. Insert</h5> Let's insert a disk! I chose mtcpget</code> because I actually forgot the contents of this image and wanted to see what I used it for. Listing the root directory using DIR</code> works without a hitch: C:\>ethflop i mtcpget Disk MTCPGET loaded (360 KiB) C:\>B: B:\>dir Volume in drive B has no label Volume Serial Number is AEA9-1EC3 Directory of B:\ MTCPGET BAT 870 01-01-80 3:34p PKT_DRVS <DIR> 05-16-12 1:52p UTILS <DIR> 05-16-12 1:53p CFG <DIR> 05-16-12 1:53p BATCH <DIR> 05-16-12 2:15p GO BAT 0 05-16-12 2:15p 6 file(s) 870 bytes 101376 bytes free B:\> </code></pre> Looks like I used this disk to bootstrap mTCP onto new machines. Displaying the contents of the MTCPGET.BAT</code> file using TYPE</code> confirms this. Good to know that accessing files works fine as well. B:\>type MTCPGET.BAT REM ECHO OFF ECHO This batch file retrieves the mTCP stack and programs using mTCP FTP. ECHO Make sure environment is unset and batch variables are set correctly before using this file. ECHO In addition, make sure the packet driver is functional. PAUSE SET SRCDRIV=A: SET PATH=%SRCDRIV%\UTILS SET DESTDRIV=C: SET MACHINE=8088 SET MTCPCFG=%SRCDRIV%\CFG\BATCHTCP.CFG SET SERVER=192.168.1.3 ECHO Switching to destination drive... %DESTDRIV% ECHO Obtaining DHCP configuration... DHCP ECHO Obtaining mTCP stack/programs... FTP %SERVER% < %SRCDRIV%\CFG\MTCP.FTP ECHO Obtaining machine-specific files... FTP %SERVER% < %SRCDRIV%\CFG\%MACHINE%.FTP ECHO Switching back to source drive... %SRCDRIV% ECHO This batch file has completed executing. ECHO It is recommended to restart the computer at this time to restore the environment. PAUSE </code></pre> Eject</h5> I didn't have much to do with MTCPGET</code>, so I soon ejected this disk. I tested that this worked by deliberately accessing a non-existent disk. I am curious that f</code>ailing</a> on the first error successfully read out a blank volume label, but even with that behavior, the eject command works fine to me: B:\>ethflop e Disk MTCPGET ejected B:\>dir Seek error reading drive B Abort, Retry, Fail?f Volume in drive B has no label Seek error reading drive B Abort, Retry, Fail?f Not ready reading drive B Abort, Retry, Fail?f Fail on INT 24 Seek error reading drive B Abort, Retry, Fail?f Current drive is no longer valid>C: </code></pre> Write Protect, Then Insert</h5> Because I live life on the edge, I decided to test whether write-protect works on one of my pristine, old MS-DOS images. If this fails, then I've lost... pretty much nothing, because this is a backup copy of the image :). I don't know why the volume label is missing 3 hex digits- I will investigate and update later. C:\>ethflop ip msdos213 Disk MSDOS213 loaded (360 KiB (write-protected)) C:\>B: B:\>mkdir FOO Write protect error writing drive B Abort, Retry, Fail?f Fail on INT 24 - FOO B:\>dir Volume in drive B is E107-A Directory of B:\ BIN <DIR> 08-27-11 12:51p COMMAND COM 16421 07-16-84 9:50a ALTCHAR SYS 431 04-04-84 8:22a AUTOEXEC BAT 23 04-04-84 8:11a 4 file(s) 16875 bytes 75776 bytes free B:\> </code></pre> Since write-protected succeeded according to the error, it looks like I saved myself an extra rsync</code>/cp</code> command to reset my MS-DOS image. And based on the modification date of BIN</code>, this image isn't pristine/original at all! Ah well... Rename</h5> Renaming a virtual disk has interesting behavior- the rename operation appears to be case insensitive, even on the Linux side! I expected the image filename on the server to be renamed with the same case that I typed it at the DOS prompt: B:\>ethflop r mydisk MYDBAK Disk MYDISK renamed to MYDBAK </code></pre> However, it looks like either ethflopd</code> or ETHFLOP.COM</code> sanitizes the image name so that only lowercase image filenames are created on the server side: wjones@DietPi:~/src/img$ ls msdos213.img mtcpget.img mydbak.img </code></pre> I did not try renaming a virtual disk that's inserted. I will update this post in the future with the behavior when I have time to test. MYDISK</code> was an image I had created in a prior ethflop session. I renamed it to MYDBAK</code> (BAK</code> as in "backup") because I want to duplicate the process of creating MYDISK</code> with a fresh new image for reasons I will describe in the next section. Speaking of which... Create An Image</h5> The most powerful feature of ethflop in my opinion is the ability to create new virtual disks to store on the server. Compared to copying floppies, creating and populating a virtual disk is much more convenient to quickly share a new disk with your other networked computers. You can even store your shiny new image image on an FTP server or BBS (those things still exist, right?) to share with other like-minded retro-enthusiasts in record time! To create an image, use the n</code> command and supply a size and disk name as arguments: B:\>ethflop n360 mydisk Disk MYDISK created (360 KiB) </code></pre> Listing the image directory contents shows that I indeed create a new floppy image. Note that the image owner and group match those of the user invoking ethflopd</code> (after setuid</code> takes effect). The permissions of mydbak.img</code> in the listing were from a previous ethflopd</code> session using sudo</code>: wjones@DietPi:~/src/img$ ls -l total 848 -rw-r--r-- 1 wjones wjones 368640 Sep 22 2012 msdos213.img -rw-r--r-- 1 wjones wjones 368640 May 16 2012 mtcpget.img -rw-r--r-- 1 root root 368640 Dec 4 13:53 mydbak.img -rw-r--r-- 1 root wjones 368640 Dec 5 13:19 mydisk.img </code></pre> After creating an image, you need to insert your shiny new disk before using it. Based on the Seek error</code> messages, I was having network trouble at the time. However, I did eventually manage to connect to the server: B:\>ethflop i mydisk ERROR: you must first eject your current virtual floppy (MSDOS213) B:\>ethflop e Disk MSDOS213 ejected B:\>ethflop i mydisk Seek error reading drive B Abort, Retry, Fail?r Seek error reading drive B Abort, Retry, Fail?r Seek error reading drive B Abort, Retry, Fail?r Seek error reading drive B Abort, Retry, Fail?f Invalid drive specification Disk MYDISK loaded (360 KiB) B:\>dir Volume in drive B has no label Volume Serial Number is 5DE9-4A1E Directory of B:\ File not found B:\> </code></pre> Once ETHFLOP.COM</code> connected to the server, I didn't have any further trouble. On a lark, I decided to try to recreate my existing MYDBAK</code> virtual disk, including the TEST</code> directory and TEST.TXT</code> file. I was curious what bytes, if any, would be different between the two images beyond timestamps. Unfortunately, I made a mistake and accidentally quoted the contents of TEST.TXT</code>. My mistake will come up again</a> a bit later. B:\>mkdir TEST Not ready writing drive B Abort, Retry, Fail?r B:\>cd TEST B:\TEST>echo "This is a test file" > TEST.TXT B:\TEST>ethflop e Disk MYDISK ejected </code></pre> Unsupported Hardware Floppy Density Emulation</h6> You can even create images of floppy densities that the hardware doesn't support! In the below snippet, I create a 1.44MB 3.5" virtual disk and populate it much the same way as the previous examples, minus the quotes this time :): C:\>ethflop n1440 hddisk Disk HDDISK created (1440 KiB) C:\>ethflop i hddisk Disk HDDISK loaded (1440 KiB) C:\>B: B:\>mkdir TEST B:\>dir Volume in drive B has no label Volume Serial Number is 5DE9-4BEB Directory of B:\ TEST <DIR> 12-05-19 1:27p 1 file(s) 0 bytes 1457152 bytes free B:\TEST>echo This is a test file > TEST.TXT B:\TEST>type TEST.TXT This is a test file B:\TEST>ethflop e Disk HDDISK ejected </code></pre> By doing a hexdump on the server (alias hd='od -Ax -t x1z'</code>), you can see the image indeed is the size of a 1.44MB disk, and includes the file I just created starting at 004400</code>. File metadata- known in FAT file system terms as the directory entry</a>- is also visible starting at 004240</code>: wjones@DietPi:~/src/img$ hd hddisk.img 000000 eb fe 90 4d 53 44 4f 53 35 2e 30 00 02 01 01 00 >...MSDOS5.0.....< 000010 02 e0 00 40 0b f0 09 00 12 00 02 00 00 00 00 00 >...@............< 000020 00 00 00 00 00 00 29 eb 4b e9 5d 4e 4f 20 4e 41 >......).K.]NO NA< 000030 4d 45 20 20 20 20 46 41 54 31 32 20 20 20 00 00 >ME FAT12 ..< 000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 0001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa >..............U.< 000200 f0 ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001400 f0 ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 001410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 002600 54 45 53 54 20 20 20 20 20 20 20 10 00 00 00 00 >TEST .....< 002610 00 00 00 00 00 00 62 6b 85 4f 02 00 00 00 00 00 >......bk.O......< 002620 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 004200 2e 20 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >. .....< 004210 00 00 00 00 00 00 62 6b 85 4f 02 00 00 00 00 00 >......bk.O......< 004220 2e 2e 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >.. .....< 004230 00 00 00 00 00 00 62 6b 85 4f 00 00 00 00 00 00 >......bk.O......< 004240 54 45 53 54 20 20 20 20 54 58 54 20 00 00 00 00 >TEST TXT ....< 004250 00 00 00 00 00 00 aa 6b 85 4f 03 00 16 00 00 00 >.......k.O......< 004260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 004400 54 68 69 73 20 69 73 20 61 20 74 65 73 74 20 66 >This is a test f< 004410 69 6c 65 20 0d 0a 00 60 a9 16 eb 00 02 2f 00 00 >ile ...`...../..< 004420 46 44 49 53 4b 20 20 20 43 4f 4d 00 00 00 00 00 >FDISK COM.....< 004430 00 00 00 00 00 00 00 60 a9 16 f1 00 d8 df 00 00 >.......`........< 004440 46 4f 52 4d 41 54 20 20 43 4f 4d 00 00 00 00 00 >FORMAT COM.....< 004450 00 00 00 00 00 00 00 60 a9 16 0d 01 8f 80 00 00 >.......`........< 004460 4d 49 52 52 4f 52 20 20 43 4f 4d 00 00 00 00 00 >MIRROR COM.....< 004470 00 00 00 00 00 00 00 60 a9 16 1e 01 f9 46 00 00 >.......`.....F..< 004480 33 43 35 58 39 50 44 20 43 4f 4d 00 00 00 00 00 >3C5X9PD COM.....< 004490 00 00 00 00 00 00 ac 80 85 26 89 13 bc 31 00 00 >.........&...1..< 0044a0 53 48 41 52 45 20 20 20 45 58 45 00 00 00 00 00 >SHARE EXE.....< 0044b0 00 00 00 00 00 00 00 60 a9 16 2a 01 90 2a 00 00 >.......`..*..*..< 0044c0 53 4d 41 52 54 44 52 56 53 59 53 00 00 00 00 00 >SMARTDRVSYS.....< 0044d0 00 00 00 00 00 00 00 60 a9 16 30 01 83 20 00 00 >.......`..0.. ..< 0044e0 53 59 53 20 20 20 20 20 43 4f 4d 00 00 00 00 00 >SYS COM.....< 0044f0 00 00 00 00 00 00 00 60 a9 16 35 01 70 34 00 00 >.......`..5.p4..< 004500 55 4e 44 45 4c 45 54 45 45 58 45 00 00 00 00 00 >UNDELETEEXE.....< 004510 00 00 00 00 00 00 00 60 a9 16 3c 01 64 36 00 00 >.......`..<.d6..< 004520 55 4e 46 4f 52 4d 41 54 43 4f 4d 00 00 00 00 00 >UNFORMATCOM.....< 004530 00 00 00 00 00 00 00 60 a9 16 43 01 90 48 00 00 >.......`..C..H..< 004540 58 43 4f 50 59 20 20 20 45 58 45 00 00 00 00 00 >XCOPY EXE.....< 004550 00 00 00 00 00 00 00 60 a9 16 4d 01 bc 3d 00 00 >.......`..M..=..< 004560 44 4f 53 4b 45 59 20 20 43 4f 4d 00 00 00 00 00 >DOSKEY COM.....< 004570 00 00 00 00 00 00 00 60 a9 16 55 01 fc 16 00 00 >.......`..U.....< 004580 44 4f 53 53 48 45 4c 4c 56 49 44 00 00 00 00 00 >DOSSHELLVID.....< 004590 00 00 00 00 00 00 00 60 a9 16 58 01 f6 24 00 00 >.......`..X..$..< 0045a0 54 43 50 20 20 20 20 20 20 20 20 10 00 00 00 00 >TCP .....< 0045b0 00 00 00 00 00 00 ee 10 24 c8 aa 04 00 00 00 00 >........$.......< 0045c0 44 4f 53 53 48 45 4c 4c 43 4f 4d 00 00 00 00 00 >DOSSHELLCOM.....< 0045d0 00 00 00 00 00 00 00 60 a9 16 63 01 02 12 00 00 >.......`..c.....< 0045e0 44 4f 53 53 48 45 4c 4c 45 58 45 00 00 00 00 00 >DOSSHELLEXE.....< 0045f0 00 00 00 00 00 00 00 60 a9 16 66 01 0e 98 03 00 >.......`..f.....< 004600 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 168000 </code></pre> I assume the density of floppy images you can create and write depends on the DOS version you're running. My 286 runs PC-DOS 5.0, which supports 1.44MB disks just fine, but I know my 286 BIOS does not support 1.44MB disks. I don't know at present how ETHFLOP.COM</code> interacts with the BIOS bookkeeping, if at all. Until I know for sure, I am using this feature with caution. In the above dump of hddisk.img</code>, the data starting at FDISK</code> and beyond appear to be legitimate directory entries from my AT's hard disk. I am not sure why ethflopd</code> is creating these entries. My guess is this behavior isn't intentional, as none of these directory entries are actually reachable; bytes 002620</code> and 004260</code> would need to be nonzero</a> to indicate more files are present in either the root or TEST</code> directories. Seeing as I'm not storing sensitive data on my AT, I'm not exactly worried about extra data being copied to my newly-created virtual disks. However, I found the behavior strange enough to be worth noting. If you're a retro enthusiast who stores sensitive data on their retro machines (I'm not gonna judge you), I would check to make sure newly-created virtual images don't accidentally copy sensitive data to the server. In the meantime, I'll look further into this behavior. Unexpected Directory Entries</h6> What follows is my attempts to flesh out why extra directory entries are created. If you don't feel like looking at hexdumps, skip</a> to the next section! The extra directory entries seem to disappear when I recreate an existing image. For example, my second (and still wrong) attempt to duplicate mydbak.img</code> from a fresh copy of mydisk.img</code> resulted in the following hexdumps: wjones@DietPi:~/src/img$ hd mydisk.img 000000 eb fe 90 4d 53 44 4f 53 35 2e 30 00 02 02 01 00 >...MSDOS5.0.....< 000010 02 70 00 d0 02 fd 02 00 09 00 02 00 00 00 00 00 >.p..............< 000020 00 00 00 00 00 00 29 1e 4a e9 5d 4e 4f 20 4e 41 >......).J.]NO NA< 000030 4d 45 20 20 20 20 46 41 54 31 32 20 20 20 00 00 >ME FAT12 ..< 000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 0001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa >..............U.< 000200 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000600 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000610 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000a00 54 45 53 54 20 20 20 20 20 20 20 10 00 00 00 00 >TEST .....< 000a10 00 00 00 00 00 00 b7 6a 85 4f 02 00 00 00 00 00 >.......j.O......< 000a20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001800 2e 20 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >. .....< 001810 00 00 00 00 00 00 b7 6a 85 4f 02 00 00 00 00 00 >.......j.O......< 001820 2e 2e 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >.. .....< 001830 00 00 00 00 00 00 b7 6a 85 4f 00 00 00 00 00 00 >.......j.O......< 001840 54 45 53 54 20 20 20 20 54 58 54 20 00 00 00 00 >TEST TXT ....< 001850 00 00 00 00 00 00 c9 6a 85 4f 03 00 18 00 00 00 >.......j.O......< 001860 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001c00 22 54 68 69 73 20 69 73 20 61 20 74 65 73 74 20 >"This is a test < 001c10 66 69 6c 65 22 20 0d 0a 00 00 00 00 00 00 00 00 >file" ..........< 001c20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 05a000 wjones@DietPi:~/src/img$ hd mydbak.img 000000 eb fe 90 4d 53 44 4f 53 35 2e 30 00 02 02 01 00 >...MSDOS5.0.....< 000010 02 70 00 d0 02 fd 02 00 09 00 02 00 00 00 00 00 >.p..............< 000020 00 00 00 00 00 00 29 7c 00 e8 5d 4e 4f 20 4e 41 >......)|..]NO NA< 000030 4d 45 20 20 20 20 46 41 54 31 32 20 20 20 00 00 >ME FAT12 ..< 000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 0001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa >..............U.< 000200 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000600 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000610 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000a00 54 45 53 54 20 20 20 20 20 20 20 10 00 00 00 00 >TEST .....< 000a10 00 00 00 00 00 00 9d 6e 84 4f 02 00 00 00 00 00 >.......n.O......< 000a20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001800 2e 20 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >. .....< 001810 00 00 00 00 00 00 9d 6e 84 4f 02 00 00 00 00 00 >.......n.O......< 001820 2e 2e 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >.. .....< 001830 00 00 00 00 00 00 9d 6e 84 4f 00 00 00 00 00 00 >.......n.O......< 001840 54 45 53 54 20 20 20 20 54 58 54 20 00 00 00 00 >TEST TXT ....< 001850 00 00 00 00 00 00 b0 6e 84 4f 03 00 16 00 00 00 >.......n.O......< 001860 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001c00 54 68 69 73 20 69 73 20 61 20 74 65 73 74 20 66 >This is a test f< 001c10 69 6c 65 2e 0d 0a 00 60 a9 16 eb 00 02 2f 00 00 >ile....`...../..< 001c20 46 44 49 53 4b 20 20 20 43 4f 4d 00 00 00 00 00 >FDISK COM.....< 001c30 00 00 00 00 00 00 00 60 a9 16 f1 00 d8 df 00 00 >.......`........< 001c40 46 4f 52 4d 41 54 20 20 43 4f 4d 00 00 00 00 00 >FORMAT COM.....< 001c50 00 00 00 00 00 00 00 60 a9 16 0d 01 8f 80 00 00 >.......`........< 001c60 4d 49 52 52 4f 52 20 20 43 4f 4d 00 00 00 00 00 >MIRROR COM.....< 001c70 00 00 00 00 00 00 00 60 a9 16 1e 01 f9 46 00 00 >.......`.....F..< 001c80 33 43 35 58 39 50 44 20 43 4f 4d 00 00 00 00 00 >3C5X9PD COM.....< 001c90 00 00 00 00 00 00 ac 80 85 26 89 13 bc 31 00 00 >.........&...1..< 001ca0 53 48 41 52 45 20 20 20 45 58 45 00 00 00 00 00 >SHARE EXE.....< 001cb0 00 00 00 00 00 00 00 60 a9 16 2a 01 90 2a 00 00 >.......`..*..*..< 001cc0 53 4d 41 52 54 44 52 56 53 59 53 00 00 00 00 00 >SMARTDRVSYS.....< 001cd0 00 00 00 00 00 00 00 60 a9 16 30 01 83 20 00 00 >.......`..0.. ..< 001ce0 53 59 53 20 20 20 20 20 43 4f 4d 00 00 00 00 00 >SYS COM.....< 001cf0 00 00 00 00 00 00 00 60 a9 16 35 01 70 34 00 00 >.......`..5.p4..< 001d00 55 4e 44 45 4c 45 54 45 45 58 45 00 00 00 00 00 >UNDELETEEXE.....< 001d10 00 00 00 00 00 00 00 60 a9 16 3c 01 64 36 00 00 >.......`..<.d6..< 001d20 55 4e 46 4f 52 4d 41 54 43 4f 4d 00 00 00 00 00 >UNFORMATCOM.....< 001d30 00 00 00 00 00 00 00 60 a9 16 43 01 90 48 00 00 >.......`..C..H..< 001d40 58 43 4f 50 59 20 20 20 45 58 45 00 00 00 00 00 >XCOPY EXE.....< 001d50 00 00 00 00 00 00 00 60 a9 16 4d 01 bc 3d 00 00 >.......`..M..=..< 001d60 44 4f 53 4b 45 59 20 20 43 4f 4d 00 00 00 00 00 >DOSKEY COM.....< 001d70 00 00 00 00 00 00 00 60 a9 16 55 01 fc 16 00 00 >.......`..U.....< 001d80 44 4f 53 53 48 45 4c 4c 56 49 44 00 00 00 00 00 >DOSSHELLVID.....< 001d90 00 00 00 00 00 00 00 60 a9 16 58 01 f6 24 00 00 >.......`..X..$..< 001da0 54 43 50 20 20 20 20 20 20 20 20 10 00 00 00 00 >TCP .....< 001db0 00 00 00 00 00 00 ee 10 24 c8 aa 04 00 00 00 00 >........$.......< 001dc0 44 4f 53 53 48 45 4c 4c 43 4f 4d 00 00 00 00 00 >DOSSHELLCOM.....< 001dd0 00 00 00 00 00 00 00 60 a9 16 63 01 02 12 00 00 >.......`..c.....< 001de0 44 4f 53 53 48 45 4c 4c 45 58 45 00 00 00 00 00 >DOSSHELLEXE.....< 001df0 00 00 00 00 00 00 00 60 a9 16 66 01 0e 98 03 00 >.......`..f.....< 001e00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 05a000 </code></pre> mydbak.img</code> was created during a previous session and contains directory entries from my AT hard disk, while mydisk.img</code> does not. When I recreate mydisk.img</code> yet again, I see the same results- no directory entries. It was at this point I realized I accidentally included quotes in TEST.TXT</code> while duplicating the layout of mydbak.img</code>. On a hunch, I deleted</a> mydisk.img</code> completely, created a new disk, and created TEST</code> and TEST.TXT</code> correctly this time. The hexdump of my new mydisk.img</code> looks like this: wjones@DietPi:~/src/img$ hd mydisk.img 000000 eb fe 90 4d 53 44 4f 53 35 2e 30 00 02 02 01 00 >...MSDOS5.0.....< 000010 02 70 00 d0 02 fd 02 00 09 00 02 00 00 00 00 00 >.p..............< 000020 00 00 00 00 00 00 29 1e 4a e9 5d 4e 4f 20 4e 41 >......).J.]NO NA< 000030 4d 45 20 20 20 20 46 41 54 31 32 20 20 20 00 00 >ME FAT12 ..< 000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 0001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa >..............U.< 000200 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000600 fd ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 >................< 000610 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 000a00 54 45 53 54 20 20 20 20 20 20 20 10 00 00 00 00 >TEST .....< 000a10 00 00 00 00 00 00 b7 6a 85 4f 02 00 00 00 00 00 >.......j.O......< 000a20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001800 2e 20 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >. .....< 001810 00 00 00 00 00 00 b7 6a 85 4f 02 00 00 00 00 00 >.......j.O......< 001820 2e 2e 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >.. .....< 001830 00 00 00 00 00 00 b7 6a 85 4f 00 00 00 00 00 00 >.......j.O......< 001840 54 45 53 54 20 20 20 20 54 58 54 20 00 00 00 00 >TEST TXT ....< 001850 00 00 00 00 00 00 10 6c 85 4f 03 00 17 00 00 00 >.......l.O......< 001860 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 001c00 54 68 69 73 20 69 73 20 61 20 74 65 73 74 20 66 >This is a test f< 001c10 69 6c 65 2e 20 0d 0a 0e 24 c8 02 00 00 00 00 00 >ile. ...$.......< 001c20 2e 2e 20 20 20 20 20 20 20 20 20 10 00 00 00 00 >.. .....< 001c30 00 00 00 00 00 00 68 0e 24 c8 00 00 00 00 00 00 >......h.$.......< 001c40 43 4f 55 4e 54 52 59 20 53 59 53 00 00 00 00 00 >COUNTRY SYS.....< 001c50 00 00 00 00 00 00 00 60 a9 16 3f 00 16 45 00 00 >.......`..?..E..< 001c60 45 47 41 20 20 20 20 20 53 59 53 00 00 00 00 00 >EGA SYS.....< 001c70 00 00 00 00 00 00 00 60 a9 16 48 00 15 13 00 00 >.......`..H.....< 001c80 4b 45 59 42 20 20 20 20 43 4f 4d 00 00 00 00 00 >KEYB COM.....< 001c90 00 00 00 00 00 00 00 60 a9 16 4b 00 3b 3b 00 00 >.......`..K.;;..< 001ca0 4b 45 59 42 4f 41 52 44 53 59 53 00 00 00 00 00 >KEYBOARDSYS.....< 001cb0 00 00 00 00 00 00 00 60 a9 16 53 00 3e 95 00 00 >.......`..S.>...< 001cc0 4e 4c 53 46 55 4e 43 20 45 58 45 00 00 00 00 00 >NLSFUNC EXE.....< 001cd0 00 00 00 00 00 00 00 60 a9 16 66 00 6c 1b 00 00 >.......`..f.l...< 001ce0 44 49 53 50 4c 41 59 20 53 59 53 00 00 00 00 00 >DISPLAY SYS.....< 001cf0 00 00 00 00 00 00 00 60 a9 16 6a 00 a5 3d 00 00 >.......`..j..=..< 001d00 45 47 41 20 20 20 20 20 43 50 49 00 00 00 00 00 >EGA CPI.....< 001d10 00 00 00 00 00 00 00 60 a9 16 72 00 e0 e5 00 00 >.......`..r.....< 001d20 48 49 4d 45 4d 20 20 20 53 59 53 00 00 00 00 00 >HIMEM SYS.....< 001d30 00 00 00 00 00 00 00 60 a9 16 8f 00 20 2d 00 00 >.......`.... -..< 001d40 4d 4f 44 45 20 20 20 20 43 4f 4d 00 00 00 00 00 >MODE COM.....< 001d50 00 00 00 00 00 00 00 60 a9 16 95 00 21 5c 00 00 >.......`....!\..< 001d60 53 45 54 56 45 52 20 20 45 58 45 00 00 00 00 00 >SETVER EXE.....< 001d70 00 00 00 00 00 00 00 60 a9 16 a1 00 e1 2e 00 00 >.......`........< 001d80 41 4e 53 49 20 20 20 20 53 59 53 00 00 00 00 00 >ANSI SYS.....< 001d90 00 00 00 00 00 00 00 60 a9 16 a7 00 3a 23 00 00 >.......`....:#..< 001da0 44 45 42 55 47 20 20 20 43 4f 4d 00 00 00 00 00 >DEBUG COM.....< 001db0 00 00 00 00 00 00 00 60 a9 16 ac 00 8a 50 00 00 >.......`.....P..< 001dc0 45 44 4c 49 4e 20 20 20 43 4f 4d 00 00 00 00 00 >EDLIN COM.....< 001dd0 00 00 00 00 00 00 00 60 a9 16 b7 00 52 31 00 00 >.......`....R1..< 001de0 45 4d 4d 33 38 36 20 20 45 58 45 00 00 00 00 00 >EMM386 EXE.....< 001df0 00 00 00 00 00 00 00 60 a9 16 be 00 5e 66 01 00 >.......`....^f..< 001e00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >................< * 05a000 </code></pre> I see directory entries yet again, and they're not even the same ones present in mydbak.img</code>! Now I'm confident this behavior isn't intentional! Delete</h5> Because I'm cautious about creating virtual disks of densities/sizes that my hardware doesn't support, I decided to delete HDDISK</code>/hddisk</code> to remove the temptation: C:\>ethflop d hddisk Disk HDDISK has been deleted </code></pre> On the server side, there are no surprises- hddisk.img</code> is gone. If someone really wants it back, they can recreate it from the hexdump! wjones@DietPi:~/src/img$ ls msdos213.img mtcpget.img mydbak.img mydisk.img </code></pre> At this point, I had tested ethflop's functionality to my satisfaction and turned off the AT for the day. I'm sure I'll be using both of them again very soon. The Verdict</h2> ethflop works pretty much flawlessly for accessing my myriad of floppy images I've accumulated over the years. As if you couldn't guess, I love seeing old hardware given a new lease on life. This applies especially when new software extends the functionality of old hardware to- or beyond- what it was capable of during its useful life. So to Mateusz Viste</a>, I sincerely thank you for writing ethflop! I have a large number of floppy images I've accumulated over the years, but not enough 5.25" floppies^{6</a> to write them all. In the past I've reused disks I've no longer needed (but had an image backup) in order to "time division multiplex" floppy disks. But I have plenty of images which only work properly as boot disks. Since I can't load ethflop during boot, I like to save my physical floppies for stuff like install or rescue disks. I think I will be using ethflop frequently from here on out for my vintage pursuits, unless I'm just in the mood to play with real, physical floppy disks :).} Future Ideas</h3> I'm not in the right frame of mind to try this now, but I wonder if it'd be possible to extend ethflop to emulate a virtual D:\ drive (I dare not try replacing drive C:\!)? Or better yet, is it possible to extend ethflop into an MSDOS block device driver instead of a TSR? What about systems with 4 floppy drives? They're rare, but the original IBM PC disk controller supports them in principle^{7</a> . Oh, and I think I can even use ethflop to implement}DISKCOPY</code></a> over the network for unimaged floppies! Mateusz Viste also wrote a networked file system</a> for DOS that I want to check out soon- that also might give me some ideas. There are many neat DOS and floppy project ideas still waiting to be pulled from the ether to give old machines new life... Footnotes</h2> 1</a> I've opened my 10BASE2 to 10BASET converter before; the conversion is done by a chip that was manufactured by National Semiconductor in the early/mid-90s. Unfortunately, its name escapes me, and I can't check in the immediate future. 2</a> I'm unsure if a "generic" packet driver exists, though I'm aware that the packet driver spec maintainer Russ Nelson</a> used to have a generic template that vendors used to create their packet drivers. 3</a> Interfacing DOS with the serial port could be a blog post in and of itself, I'm afraid. It's very easy to get a very helpful Write failure</code> or General failure</code> when accessing the serial port (e.g. COPY COM2 FOO.TXT</code> or CTTY COM2</code>) just by breathing wrong. What best matches my experience is that DOS uses flow control</a> for sending but not receiving (Source</a>). Disabling flow control in your serial terminal or transfer program is a safe option. 4</a> I don't remember the exact details right now (and will check later), but I had an old 8-bit ISA CNet NIC that required a few kB of Upper Memory</a>. The manufacturer-set default switches for the memory area- yes, this had to be set manually!- conflicted with the memory area commonly used for hard drive controller (segment 0xc800</code>). This was even mentioned in the manual I got with the card, but of course I didn't read it first :). 5</a> Originally, the IBM PC had provisions for 4 floppy drives, though I don't remember how DOS assigns drive letters in this case, especially in the case where hard disks exist as well. This might be an interesting addition to ethflop to keep Drives A:\ and B:\ free. 6</a> I think I have about 50 5.25" floppies overall? Over the years I've accumulated approximately 30 Double Density (360kB) and 20 High Density (1.2MB) disks. Sadly, it is easier for me to write and read 5.25" disks than 3.5" disks ever since my USB floppy drive broke... 7</a> As far as I'm aware, no commercial product was ever released that used the DB37 connector on the original IBM PC floppy controller ISA card. However, the pinout matches the more traditional 34-pin edge connector, so an adapter isn't difficult. Porting a New Board To Migen 2017-09-25T00:00:00+00:00 Porting a New Board To Migen</h1> Taps mic Is this thing on? So it's really been 11 months since I last wrote something? I really need to change that! I'm going to attempt to write smaller articles in between my larger, more involved ones to keep me motivated to keep writing. So today, I'm going to write about a simpler topic I meant to discuss last year, but never got around to it: How to get started with Migen</a> on your Shiny New FPGA Development Board! This post assumes you have previous experience with Python 3 and experience synthesizing digital logic on FPGAs with Verilog. However, no Migen experience is assumed! Yes, you can port your a development board to Migen without having used Migen before! What Is Migen- And Why Use It?</h2> Migen is a Python library (framework, maybe?) to build digital circuits either for simulation, synthesis on an FPGA, or ASIC fabrication (the latter of which is mostly untested). Migen is one</a> of</a> many</a> attempts to solve some deficiences with Verilog and VHDL, the languages most commonly used in industry to develop digital integrated circuits (ICs). If necessary, a designer can use and import Verilog/VHDL directly into their designs using one of the above languages too. All the languages above either emit Verilog or an Intermediate Representation</a> that can be transformed into Verilog/VHDL. I can think of a few reasons why: FPGA synthesis/ASIC front-ends are typically proprietary, can't be readily modified, and mostly only support Verilog and VHDL as input.</li> Many issues with Verilog, such as synthesis-simulation behavior mismatch, don't occur if Verilog can be emitted in a controlled manner, as is done with the above languages.^{1</a> </li>} From my own experience looking at yosys, which can be used as the front- end Verilog compiler to an open-source synthesis toolchain</a>, there is little gain to targeting a file format meant for FPGA synthesis (or ASIC fabrication) directly from a higher-level language. Targeting the synthesis file format directly must be shown to be bug-free with respect to having generated Verilog and then generating the synthesis file format from the Verilog; this is nontrivial.</li> </ol> The Migen manual</a> discusses its rationale for existing, but the important reasons that apply to me personally are that Migen: Avoids simulation-synthesis behavior mismatches in generated output.</li> Makes it easy to automate digital logic design idioms that are tedious in Verilog alone, such as Finite-State Machines and Asynchronous FIFOs.</li> Prevents classes of bugs in Verilog code, such as ensuring each signal has an initial value and unused cases in switch</code> statements are handled.</li> Handles assignments satisfying "same sensitivity list, multiple always blocks" by concatenating them all into a single always</code> block, whereas Verilog forbids this for synthesis. In my experience, this increases code clarity.</li> </ul> As an added bonus, I can write a Migen design once and, within reason, generate a bitstream of my design without needing a separate User Constraints File (UCF) for each board that Migen supports. This facilitates design reuse by others who may not have the same board as I do, but has a new board with the required I/O peripherals anyway. For the above reasons, I am far more productive writing HDL than I ever was writing Verilog.^{2</a> } Choice of Tools Disclaimer</h3> Of the linked languages above, I have only personally used Migen. Migen was the first Verilog alternative I personally discovered when I started using FPGAs for projects again in 2015 (after a 3 year break). When I first saw the typical code structure of a Migen project, I immediately felt at home writing my own basic designs, and could easily predict which Migen code would emit which Verilog constructs without reading the source. In fact, I ported my own Shiny New FPGA Development Board I just bought to Migen before I even tested my first Migen design! Because of my extremely positive first impressions, and time spent learning Migen's more complicated features, I've had little incentive to learn a new HDL. That said, I maintain it is up to the reader to experiment and decide which HDL works best for their FPGA/ASIC projects. I only speak for my own experiences, and the point of me writing this post is to make porting a new board to Migen easier than when I learned how to do it. The code re-use aspect of Migen is important to me, and when done correctly, a port to a new board is very low-maintenance. Leveraging Python To Build FPGA Applications</h2> To motivate how to port a new development board, I need to show Migen code right now. If you haven't seen Migen before now, don't panic! I'll briefly explain each section: from migen import * from migen.build.platforms import icestick class Rot(Module): def __init__(self): self.clk_freq = 12000000 self.ready = Signal() self.rot = Signal(4) self.divider = Signal(max=self.clk_freq) self.d1 = Signal() self.d2 = Signal() self.d3 = Signal() self.d4 = Signal() self.d5 = Signal() ### self.comb += [j.eq(self.rot[i]) for i, j in enumerate([self.d1, self.d2, self.d3, self.d4])] self.comb += [self.d5.eq(1)] self.sync += [ If(self.ready, If(self.divider == int(self.clk_freq) - 1, self.divider.eq(0), self.rot.eq(Cat(self.rot[-1], self.rot[:-1])) ).Else( self.divider.eq(self.divider + 1) ) ).Else( self.ready.eq(1), self.rot.eq(1), self.divider.eq(0) ) ] </code></pre> If you've never seen Migen code before, and/or are unfamiliar with its layout, I'll explain some of the more interesting points here in comparison to Verilog: from migen import * from migen.build.platforms import icestick </code></pre> Migen by default exports a number of primitives to get started writing Modules</code>. Other Migen constructs, such as a library of useful building blocks (migen.genlib</code>) and Verilog code generation (migen.fhdl.verilog</code>) must be imported manually. migen.build.platforms</code> contains all the FPGA development platforms Migen supports; the goal of this blog post will be to reimplement the iCEstick</a> development board for use from Migen. </li> class Rot(Module): </code></pre> A Module</code> is the basic unit describing digital behavior in Migen. Connections between Module</code>s are typically made by declaring instance variables that can be shared between Module</code>s, and using submodules</code>. Submodule syntax is described in the manual, and is required to share connections between modules. </li> self.ready = Signal(1) </code></pre> A Signal</code> is the basic data type in Migen. Unlike Verilog, which requires keywords to distinguish between data storage (reg</code>) and connections/nets (wire</code>), Migen can infer from a signal's usage whether or not the signal stores data. The 1</code> indicates the signal is one-bit wide. </li> self.divider = Signal(max=self.clk_freq) </code></pre> In addition to providing a bit width, I can tell Migen the maximum value a Signal</code> is expected to take, and Migen will initialize its width to log2(max) + 1</code>. However, nothing prevents the signal from exceeding max when run in a simulator or FPGA. </li> ### self.comb += [j.eq(self.rot[i]) for i, j in enumerate([self.d1, self.d2, self.d3, self.d4])] </code></pre> By convention, data-type declarations are separated from code describing how code should behave using ###</code>. Migen statements relating connections of Signal</code>s and other data types must be appended to either a comb</code> or sync</code> attribute. Connections are typically made using the eq</code> function of Signal</code>s. Python's slice notation accesses individual or subsets of bits of a Signal</code>. comb</code>, or combinationial, statements are analogous to Verilog's assign</code> keyword or combinational always @(*)</code> blocks, depending on the Migen data type/construct. In combinational logic, the output immediately changes in response to a changing or just-initialized) input without any concept of time (in theory). </li> self.sync += [ If(self.ready, If(self.divider == int(self.clk_freq) - 1, self.divider.eq(0), self.rot.eq(Cat(self.rot[-1], self.rot[:-1])) ).Else( self.divider.eq(self.divider + 1) ) ).Else( self.ready.eq(1), self.rot.eq(1), self.divider.eq(0) ) ] </code></pre> By contrast, appending to the Migen sync</code>, or synchronous, attribute emits Verilog code whose output only changes in response to a positive edge clock transition of a given clock, using the following syntax: module.sync.my_clk += [...]</code>. If the clock is omitted, the clock defaults to sys</code>. In synchronous/sequential logic, outputs do not change immediately in response to a changing or just- initialized input; the output only registers a new value based on its input signals in response to a low-to- high transition of another, usually periodic, signal. Migen only always @(posedge clk)</code> blocks, so a negedge</code> clock must be created by inverting an existing clock, such as self.comb += [neg_clk.eq(~pos_clk)]</code>. As one might expect, If().Elif().Else()</code> is analogous to Verilog if</code>, else if</code>, and else</code> blocks. Migen will generate the correct style of always</code> block to represent signal transitions regardless of whether the If()</code> statement is part of a comb</code> or sync</code> block. </li> </ul> I omitted discussing any Migen data types, input arguments to their constructors, other features provided by the package, and common code idioms that I didn't use above for the sake of keeping this blog post on point. Most of the user-facing features/constructors are documented in the user manual. I can discuss features (and behavior) not mentioned in the manual in a follow-up post. Adding a New Board</h2> The above code</a> was adapted from the rot.v</a> example of Arachne PNR. In words, the above code turns on an LED, counts for 12,000,000 clock cycles, then turns off the previous LED and lights another of 4 LEDs; after 4 LEDs the cycle repeats. A fifth LED is always kept on as soon as the FPGA loads its bitstream. Our goal is to get this simple Migen design to work on a new FPGA development board, walking through the process. Since this example is tailored to the iCE40 FPGA family, I'm choosing to port the iCEstick board to Migen... which already has a port... Interactive Porting</h3> My original intention was to write this blog post while I was creating the icestick.py</code> platform file to be committed into migen.build</code>. Unfortunately, at the time, Migen did not have any support for targeting the IceStorm synthesis toolchain.^{3</a> So I ended up}implementing</a> the IceStorm backend and doing my blog post as intended went by the wayside while debugging. That said, I'm going to attempt to simulate the process of adding a board from the beginning. I will only assume that the IceStorm backend to Migen exists, but the icestick.py</code> board file does not. We Need a Constraints File</h3> Before I can start writing a board file for iCEstick, we need to know which FPGA pins are connected to what. For many boards, the manufacturer will provide a full User Constraints File (UCF) with this information. For demonstrative purposes^{4</a> however, I will examine the schematic diagram of iCEstick instead to create my Migen board file. This can be found in a file provided by Lattice at a changing URL called "icestickusermanual.pdf".} We need to know the format of FPGA pin identifiers that Arachne PNR, the place-and-route tool for IceStorm, will expect as well. The format differs for each of the FPGA manufacturers and even between FPGA families of the same manufacturer; Xilinx, for example uses [A-Z]?[0-9]*</code>, as does the Lattice ECP3 family. Fortunately, IceStorm uses pin numbers that correspond to the device package, and these are easily visible on the schematic: One side (port) of connections of the iCE40 FPGA to iCEstick peripherals. In this image, LED, IrDA, and one side of 600 mil breadboard-compatible breakout connections can be seen.</figcaption> </figure> If we examine the schematic and user manual, we will find the following peripherals: 5 LEDs</li> 2x6 Digilient PMOD connector</li> IrDA transceiver</li> SPI Flash</li> FT2232H UART (one channel- the other is used for programming)</li> 16 Additional I/O pins</li> </ul> We might not have an actual full constraints file to work with due to how arachne-pnr</code> works, but we have all the information to create a Migen board file anyway, since we have the schematics. Armed with this information, we can start creating a board file for iCEStick. Anatomy of a Migen Board File</h3> Relative to the root of the migen package, migen places board definition files under the build/platforms</code> directory. All paths in this section, unless otherwise noted are relative to the package root. Platform Class</h4> from migen.build.generic_platform import * from migen.build.lattice import LatticePlatform from migen.build.lattice.programmer import IceStormProgrammer class Platform(LatticePlatform): default_clk_name = "clk12" default_clk_period = 83.333 def __init__(self): LatticePlatform.__init__(self, "ice40-1k-tq144", _io, _connectors, toolchain="icestorm") def create_programmer(self): return IceStormProgrammer() </code></pre> A board file consists of the definition of Python class</code> conventionally named Platform</code>. Platform</code> should inherit from a class defined for each supported FPGA manufacturer. As of this writing, Migen exports AlteraPlatform</code>, XilinxPlatform</code>, and LatticePlatform</code>, and more are possible in the future. Vendor platforms are defined in a subdirectory under build</code> for each vendor, in the file platform.py</code>. Each FPGA vendor in turn inherits from GenericPlatform</code>, which is defined in build/generic_platform.py</code> and exports a number of useful methods for use in Migen code (I'll introduce them as needed). The GenericPlatform</code> constructor</a> accepts the following arguments: device</code>- A string indicating the FPGA device on your board.^{5</a> The string format is vendor-toolchain specific; in the case of IceStorm, the format is currently "ice40-{1k,8k}-{package}"</code>.</li>} io</code>- A list of tuples of a specific format which I'll describe shortly. The list represents all non-user-expandable I/O resources on your current board. For instance, LEDs, SPI flash, and ADC connections to your FPGA would be placed in the io</code> list.</li> connectors</code>- A list of tuples with a specific layout, which I'll describe shortly. The list represents user-expandable I/O that by default is not connecte to any piece of hardware, [Pmod](https://en.wikipedia.org/wiki/ Pmod_Interface, Pmod) headers.</li> name</code>- I don't know what this input argument does exactly. Like other places in Migen with a name</code> input argument, it's meant to control how variable names are generated in the Verilog output. I don't believe it's used by any board file, so I'm ignoring it.</li> toolchain</code>- The same FPGA vendor can have multiple software suites for their FPGA families, or third-party toolchains can exist. For instance, Xilinx has both the End-of-Life ISE Toolchain and Vivado software available. Additionally, Lattice provides the Diamond toolchain for their higher-end FPGAs while Project IceStorm is an unaffialited open-source synthesis flow for the iCE40 family. Thus, the vendor platform constructors also supply</a> a toolchain</code> keyword argument to choose which toolchain to eventually invoke. In the case of iCEStick, we use Migen's icestorm</code> backend.</li> </ul> A Platform</code> class definition should also define the class variables default_clk_name</code> and default_clk_period</code>, which are used by GenericPlatform</code>. default_clk_name</code> should match the name of a resource in the io</code> list that represents a clock input to the FPGA. default_clk_period</code> is used by vendor-specific logic in Migen to create a clock constraint in nanoseconds for default_clk_name</code>. The default clock is associated with the sys</code> clock domain for sync</code> statements. Lastly, the create_programmer</code> function should return a vendor-specific programmer. Adding a programmer is beyond the scope of this article. If a board can support more than one programming tool, the convention is to return a programmer based on a programmer</code> class variable</a> for the given board. This function can be omitted if no programmer fits, or one can be created on-the-fly using GenericPlatform.create_programmer</code>. Finalization</h4> Some platforms Migen supports, such as the LX9 Microboard</a>, have a do_finalize</code> method. Finalization in Migen allows a user to defer adding logic to their design until overall resource usage is known. In particular, LX9 Microboard has an Ethernet peripheral, and the Ethernet clocks should use separate timing constraints from the rest of the design. The linked code detects whether the Ethernet peripheral was used using lookup_request("eth_clocks")</code> from GenericPlatform</code>, and adds appropriate platform constraints to the current design to be synthesized if necessary. If the Ethernet peripheral was not used in the design, the extra constraints are not added, and the ConstraintError</code> from lookup_request</code> is ignored. Finalization operates on an internal Migen data structure called Fragment</code>s. Fragment</code>s require knowledge of Migen internals to use properly, so for the time being I suggest following the linked example if you need to add constraints conditionally. Of course, timing constraints and other User Constraints File data can be added at any point in your design manually using add_period_constraint</code> and add_platform_command</code> respectively, both from GenericPlatform</code>. iCEStick does not have any peripherals which need special constraints, and only a single clock; Migen will automatically add a constraint for the default clock. More importantly, in the case of IceStorm/iCEStick, only a global clock constraint is supported due to limitations in specifying constraints. Therefore, I omit the do_finalize</code> method for the iCEStick board file. However, one use I have found for do_finalize</code> in platforms compatible with IceStorm is to automatically instantiate pins with pullup resistors enabled. This gets around the limitations of Arachne PNR's constraints file format without needing to instantiate Verilog primitives directly in the top level of a Migen source file, and I can show code upon request. I/O and Connectors</h4> After defining a Platform</code> class for your board, all you need to do is fill in a list of _io</code> and _connectors</code> in your board file, pass them into your Platform</code>'s vendor-specific base class constructor, and Migen will take care of the rest! As I stated before, io</code> and connectors</code> input arguments to the vendor-specific platform constructor are lists of tuples with a specific format. Let's start with an I/O tuple: io_name, id, Pins("pin_name", "pin_name") or Subsignal(...), IOStandard("std_name"), Misc("misc")) </code></pre> An io_name</code> is the name of the peripheral, and should match the string passed into the request</code> function to gain access to the peripheral's signals from Migen. id</code> is a number to distinguish multiple copies of identically- functioning peripherals, such as LEDs. For simple peripherals, Pins</code> is a helper class which should contain strings corresponding to the vendor- specific pin identifiers where the peripheral connects to the FPGA; in the case of IceStorm, there are just the pin numbers as defined on the package pinout. I will discuss Subsignal</code>s in the next paragraph. These tuple entries are used to create inputs and output names in the Migen-generated Verilog, and provide a variable-name to FPGA pin mapping in a Migen- generated User Constraints File (UCF) Without going into excess detail^{6</a> , Subsignals</code> are a helper class for resources that use FPGA pins which can be seperated cleanly by purpose. The inputs to a Subsignal</code> constructor are identical to an I/O tuple entry, except with id</code> omitted. The net effect for the end user is that a resource is encapsulated as a class whose Migen Signals</code> are accessed via the class' members, i.e. comb += [my_resource.my_sig.eq(5)]</code>. This is known as a Record</code> in Migen. Records</code> also come with a number of useful}methods</a> for constructing Migen statements quickly. Think of them as analogous to C structs</code>. It is up to your judgment whether an I/O peripheral should use Subsignals</code>, but in general, I notice that Migen board files make heavy use of them. The remaining inputs to an I/O tuple entry are optional. IOStandard</code> is another helper class which contains a toolchain-specific string that identifies which voltages/logic standard should use. And lastly, the Misc</code> helper class contains a space-separated string of other information that should be placed into the User Constraints File along with IOStandard</code>. Such information includes slew rate</a> and whether pullups should be enabled. These are in fact currently ignored in the IceStorm toolchain, but for my own reference I have filled them in as necessary. A connector tuple is a bit simpler: (conn_name, "pin_name, pin_name, pin_name,...") </code></pre> conn_name</code> is analogous to io_name</code>. The second element of a connector tuple is a space-separated string of pin names matching the vendor's format which indicates which pins on the FPGA are associated with that particular connector. Ideally, the pins should be listed in some order that makes sense for the connector. By default, pins that are associated with connectors are not exposed by the Platform</code> via the request</code> method. Instead, a user needs to notify the platform that they wish to use the connector as extra I/O using the GenericPlatform.add_extension</code> method. Here is an example adding a PMOD I2C peripheral</a> using add_extension</code>: my_i2c_device = [ ("i2c_device", 0, Subsignal("sdc", Pins("PMOD:2"), Misc("PULLUP")), Subsignal("sda", Pins("PMOD:3"), Misc("PULLUP")) ) ] plat.add_extension(my_i2c_device) plat.request("i2c_device") </code></pre> Note that adding a peripheral using add_extension</code> is similar to adding a peripheral to the io</code> list, except that the Pins()</code> element takes on the form "conn_name:index"</code>. conn_name</code> should match a tuple in the connectors</code> list, and index</code> is a zero-based index into the string of FPGA pins associated with the connector. This allows you to create peripherals on-the-fly that are (in theory) board and vendor-agnostic. With the last concepts out of the way, let's jump right into creating the _io</code> and _connectors</code> list for our Platform. Each listed peripheral is implied to be a tuple inside _io = [...] or _connectors = [...])</code>: ("user_led", 0, Pins("99"), IOStandard("LVCMOS33")), ("user_led", 1, Pins("98"), IOStandard("LVCMOS33")), ("user_led", 2, Pins("97"), IOStandard("LVCMOS33")), ("user_led", 3, Pins("96"), IOStandard("LVCMOS33")), ("user_led", 4, Pins("95"), IOStandard("LVCMOS33")), </code></pre> user_led</code>s are simple peripherals found on just about every development board; iCEStick has 5 of them, all identical in function (but the 5th one is green!). Resources with identical function that differ only in a pin should each be declared in their own tuple, incrementing the id</code> index. Resource signal names are by convention; if a resource does not yet exist, it's up to you what you want to name the resource. However, I suggest looking at other board files for prior examples. user_led</code>, user_btn</code>, serial.rx</code>, serial.tx</code>, spiflash</code>, and audio</code> are all commonly-used I/O names used between board files. ("serial", 0, Subsignal("rx", Pins("9")), Subsignal("tx", Pins("8"), Misc("PULLUP")), Subsignal("rts", Pins("7"), Misc("PULLUP")), Subsignal("cts", Pins("4"), Misc("PULLUP")), Subsignal("dtr", Pins("3"), Misc("PULLUP")), Subsignal("dsr", Pins("2"), Misc("PULLUP")), Subsignal("dcd", Pins("1"), Misc("PULLUP")), IOStandard("LVTTL"), ), </code></pre> Next we have another common peripheral- a UART/serial port. A UART peripheral makes sense to divide using Subsignal</code>s, since each pin has a distinct purpose. Although in practice most users will only use rx</code> and tx</code>^{7</a> , I include all possible pins just in case. I don't remember why I included PULLUP</code> as constraints information for a majority of pins. Note that it's perfectly okay to associate a constraint with all Subsignal</code>s at once, as I do for the (unused) IOStandard</code>.} ("irda", 0, Subsignal("rx", Pins("106")), Subsignal("tx", Pins("105")), Subsignal("sd", Pins("107")), IOStandard("LVCMOS33") ), </code></pre> The infrared port on iCEStick is another serial port, sans most of the control signals. I omit the optional I/O tuple/Subsignal</code> entries here, and define the IOStandard</code> similarly to the previous serial</code> peripheral. ("spiflash", 0, Subsignal("cs_n", Pins("71"), IOStandard("LVCMOS33")), Subsignal("clk", Pins("70"), IOStandard("LVCMOS33")), Subsignal("mosi", Pins("67"), IOStandard("LVCMOS33")), Subsignal("miso", Pins("68"), IOStandard("LVCMOS33")) ) </code></pre> spiflash</code> and its Subsignal</code> have standardized, self-explanatory names. I suggest using these signal names when appropriate for all peripherals connected via an SPI bus</a>. I don't remember why I added the IOStandard</code> per-signal instead of all-at- once here, but the net effect would be the same either way if IceStorm made use of IOStandard</code>. ("clk12", 0, Pins("21"), IOStandard("LVCMOS33")), </code></pre> Clock signals should be included as well, with at least one clock's io_name</code> matching the default_clk_name</code>. Migen may automatically request</code> a clock for your design if certain conditions are met; for now, assume you don't have to request</code> the clock. ("GPIO0", "44 45 47 48 56 60 61 62"), ("GPIO1", "119 118 117 116 115 114 113 112"), ("PMOD", "78 79 80 81 87 88 90 91"), </code></pre> And lastly we have the connectors. The connector pins for GPIO0-1</code> and PMOD</code> are ordered in increasing pin order, which matches the order they are laid out on their respective connectors. This is a happy coincidence. Make sure to check your schematic and declare FPGA pins in connector order, not the other way around! Building Our Design For Our "New" Board</h2> Now that we have created our board file, we now need to write the remaining logic to attach the board's I/O to our Rot top level</a>. Assuming the Rot</code> module is already defined, we can create a script that will synthesize our design like so: if __name__ == "__main__": plat = icestick.Platform() m = Rot() m.comb += [plat.request("user_led").eq(l) for l in [m.d1, m.d2, m.d3, m.d4, m.d5]] plat.build(m, run=True, build_dir="rot", build_name="rot_migen") plat.create_programmer().flash(0, "rot/rot_migen.bin") </code></pre> Once again, I will explain each line. It should be appended to the Rot</code> top level and then run using your Python 3 interpreter: plat = icestick.Platform() m = Rot() </code></pre> We create our platform and our Rot</code> module here. plat</code> contains a number of useful helper methods that help customize what is eventually sent to the synthesis flow. </li> m.comb += [plat.request("user_led").eq(l) for l in [m.d1, m.d2, m.d3, m.d4, m.d5]] </code></pre> This list comprehension is how we actually connect our I/O to the LED rotation module. plat.request("user_led")</code> will create a Migen Signal</code> or Record</code> which can be used to assign to Signals</code> and Records</code> in an existing Module</code>. GenericPlatform</code> does bookkeeping to figure out which Signals</code> should be considered I/Os to/from the generated Verilog top level (and consequently, mapped to a User Constraints File). Repeatedly calling request</code> for a given resource name will return a new Signal</code> or Record</code> of the same type, throwing a ConstraintError</code> exception if there are no more available resources </li> plat.build(m, run=True, build_dir="rot", build_name="rot_migen") </code></pre> The GenericPlatform.build</code> function will emit Verilog output, a User Constraints File specific to your design, a batch file or shell script to invoke the synthesis flow, and any other files required as input to the synthesis toolchain. Optionally, Migen will invoke the generated script to create your design if input argument run</code> is true. build_dir</code> is self- explanatory, and build_name</code> controls the filename of generated files. I pass in my top level Rot</code> Module</code> to the build</code> function. GenericPlatform</code> has internal logic to detect which signals were declared as I/O in the board file while generating Verilog input and output arguments. </li> plat.create_programmer().flash(0, "rot/rot_migen.bin") </code></pre> This line is optional, but if included, Migen will invoke the iceprog</code> FTDI MPSSE programmer for iCEStick automatically after creating a bitstream using IceStorm. The first input argument is the start address to start programming, and the second argument is the name of the output bitstream. The name can be inferred from build_name</code> and the toolchain's default extension for bitstreams. </li> </ul> If all goes well, and you were following along, you should now have a blinking LED example on your iCEStick! The final iCEStick platform board file is here</a>, which you can use as a reference, and I've made the Rot</code> top level available as a gist</a>. Take a look at the output files Migen generated, including the output Verilog and User Constraints File, to get a feel of how our "shiny new" board file was used! Happy Hacking!</h2> If you have read up to this point, you now have some grasp on the Migen framework, and can now start using Migen in your own designs on your own development (or even deployed) designs! Porting Migen to support your design takes relatively little effort, and you will quickly make up the time spent porting. Besides automating HDL idioms that are tedious to write by hand (such as FSMs), and generating Verilog that is free of certain bug classes, Migen saves time as it automates generating input files and build commands for your synthesis toolchain, and then invoking the synthesis flow automatically. Additionally, if you write your top-level Module</code> and glue code correctly, you can have a single HDL design that runs on all platforms that Migen supports, even between vendors, with much less effort than is required to do the same in Verilog alone!^{8</a> } Migen is certainly a step in the right direction for the future of hacking with FPGAs, and I hope you as the reader give it a try with your Shiny New Development Board like I did, and see whether your experiences were as positive as mine. Acknowledgements</h2> I'd like to thank Sébastien Bourdeauducq</a>, the primary architect and maintainer of Migen, for looking over an initial version of this post and offering feedback. Footnotes</h2> 1</a> For better or worse, emitting Verilog requires someone intimately familiar with the Verilog specification. Like all good specifications, it is terse, and requires a lot of memorization. But both Yosys and Migen are by and large the work of one individual each, so it can be done. 2</a> In the interest of fairness, fusesoc</a> exists now to alleviate the burdern of Verilog code-sharing. I was unaware of its existence when I started using FPGAs again. I think Migen build i ntegration would be an interesting project; in general, I find setting up a board-agnostic Migen design easier than w/ FuseSoC, but importing Verilog code as a package is still incredibly useful. 3</a> I erroneously assumed that because the Xilinx backend code can use the yosys Verilog compiler that there was support for support for yosys and by extension IceStorm in the Lattice backend. Not having had any Lattice FPGAs before iCEstick (yes, I jumped on the FOSS FPGA toolchain bandwagon), I never actually bothered to check beforehand! 4</a> Historically, I've found that IceStorm Verilog code samples only define the pins their designs actually use. Unlike Quartus or ISE, Arachne PNR will error out instead of warn if constraints that are defined aren't actually used. Migen doesn't have this issue because it will only generate constraints that are actually used for Arachne PNR. 5</a> Each Platform</code> consists of a single FPGA and attached peripherals. I assume a board with multiple FPGAs should implement a Platform</code> for each FPGA in a single board file. 6</a> Subsignals also modify</a> how signal names are generated for the remainder of the synthesis toolchain. 7</a> iCEStick disappoints me in that the engineers wired nearly all the connections required to use the FT2232H in FT245 queue mode, which is much faster than a serial port; most dev boards do not bother connecting anything besides serial TX and RX, and possibly RTS/CTS. But alas, there's still not enough control connections to use queue mode. 8</a> Portability of code gives me joy, and HDL portability is no exception. NMOS IC Reverse Engineering 2016-10-12T00:00:00+00:00 NMOS IC Reverse Engineering</h1> Recently, I was tricked</del> talked</ins> into examining the die of an integrated circuit (IC)- the YM2151, a single-chip FM synthesizer. I like these old Yamaha chips, so I don't really mind doing it, but it's definitely a long term project that I expect to take months. However, one doesn't need to RE a significant portion of an IC to understand the basics of RE. Information about a chip's operation can be gleamed by examining small units, and predicting their relation to each other. Information on doing IC reverse-engineering is still kind of limited, although projects like siliconpr0n</a> and whatever Ken Shirrif</a> is working on at any given time are changing that. Since I am learning how to RE ICs, I decided to document how I decoded a small ROM in the YM2151 that I suspect is being used as part of the control state machine. This small ROM demonstrates the basics of REing ICs, including: Separating the various layers of an IC die.</li> Mapping ROM inputs and outputs.</li> Manually reading out the ROM contents.</li> </ul> Obtaining a Die Image</h2> Before we can examine an IC die, we have to actually digitally capture an image of the die. I will not be discussing this in detail, but getting a die image typically involves: Removing the IC package with corrosive chemicals, called decap.</li> Taking a number of pictures using a digital camera with a microscope.</li> Stitching all the individual images together to produce a full map of the IC die with individual transistors visible.</li> </ul> I really don't have access to the equipment to do this even at a small scale, but luckily this work was done previously in the case of YM2151 (20x magnification). I defer to Travis Goodspeed's article in section 9 of POC||GTFO #4</a>) if interested in doing decap yourself. Die image of YM2151 at 20x magnification, full die. The full-size image is 617MB Source</a>.</figcaption> </figure> What is NMOS?</h2> The YM2151 uses an NMOS process</a>. The NMOS logic family uses n-type metal oxide semiconductor field-effect transistors (MOSFETs) to create digital logic gates. MOSFETs are four- terminal devices that either permit or prevent current from flowing between two of the terminals, called the drain and source, depending on the voltage of the third terminal, called the gate, relative to either the drain or source. The fourth terminal, called the body, is negligible for now. MOSFET Schematic Symbols</h3> The below picture shows two different types of MOSFETS: n-type depletion and n-type enhancement mode. On each MOSFET, the center terminal attached to the left column is the gate input. The source and drain are the bottom and top terminals, respectively. Each MOSFET has the source connected to a fourth terminal, which in turn connects to an arrow pointing inward to the right column, indicating n-type MOSFETs. Put simply, the arrow direction corresponds to device polarity at a specific area within the MOSFET. The source-body connection is a side effect of the MOSFET symbol I used, and we can ignore it. A segmented right column represents enhancement mode, and a solid right column represents depletion mode. A depletion and enhancement mode n-type MOSFET circuit and an equivalent circuit. The depletion mode MOSFET acts as a resistor, and the enhancement mode MOSFET as a switch. A quick rationale on the schematic symbol is given in the above paragraph. On is 5V, off 0V.</figcaption> </figure> A MOSFET is a symmetrical device, so drain and source can be labeled arbitrarily. However, according to Sedra and Smith, drain is by convention always at a higher voltage than source. Quick "Theory" of MOSFETs</h3> "n-type" refers to the properties of the silicon that carries current through a MOSFET. For the purposes of this blog post, I need not go into its properties. In hand-calculations, a MOSFET has at least three modes of operation</a>. I'm not interested in doing simulation in this blog post (maybe in the future I will!), but for completeness, the regions are: Cutoff, no conduction from drain to source.</li> Triode, linear region, acts like a resistor.</li> Saturated, increasing gate voltage does not increase current.</li> </ul> Both enchancement mode and depletion mode MOSFETs are used in the NMOS logic family. Depletion-mode MOSFETs conduct current even when the gate and source is at the same voltage; the gate must have a lower voltage than the source for cutoff to occur. They are commonly, but not exclusively, used as pullup resistors, operating in the linear region and always on. On the other hand, enhancement-mode MOSFETs are used to implement logic gates by either allowing current to pass or not. They operate in cutoff and saturation. The gate must be at a higher voltage than the source to conduct current. Voltage drop tends to be negligible in the enhancement-mode MOSFETs; see ratioed logic in the previous link for more information. With the above two paragraphs in mind, here is an exercise: How would you implement an NMOS NOR gate (hint: MOSFETs in parallel)? NAND gate (hint: MOSFETs in series)? Inverter (NOT) (hint: Look carefully at my MOSFET image)? Notice how gates with inverters are easiest to implement? I wonder if that's a reason active-low inputs and outputs are so common in these chips? Am I Dealing With NMOS?</h3> How does one detect an NMOS chip? To be honest, I was told ahead of time that the YM2151 uses an NMOS process. The year that this chip was first produced (1984-5) is also a hint. Howvever, when compared to CMOS die</a> of a similar-era Yamaha chip, I notice a few differences: Only one size of via in a CMOS die, many in NMOS.</li> No obvious indication of pullup MOSFETs in CMOS, prevalent in NMOS.</li> The CMOS die is more neatly organized, compared to NMOS.</li> </ul> Right now, the above list probably isn't all that meaningful :P. I'll discuss what I mean in the next section. As you may expect, my method of analysis only works for ICs made with an NMOS process. However, this is still useful for preserving many old chips where fully emulating their behavior is desired (YM2151) or even required (security chips) to preserve hardware. IC Layers</h2> I decided to digitize (or vectorize) a ROM at the top of the chip, approximately one third of the length longways. Section of the YM2151 I intend to vectorize, before we begin any work.</figcaption> </figure> ICs- well, not cutting-edge ones anyway- tend to be made by applying planar layers of conducting and semiconductor material. Therefore, it tends to be safe to represent each layer of material as a 2d layer in a vector image, not worrying about layer depth or wells that would be apparent in a 3d cutaway. Each layer can thus be inferred and by examining intersections and outlines left over during decap. I can tell the above is a ROM from the equal-width strips of metal (remember, metal tends to be consistently colored) and the circular "holes" distributed throughout the metal strips. These "holes" are properly referred to as "vias". Vias are drilled holes, forming a connection with any layer that intersects at their location. Additionally, there exist buried contacts that directly connect (typically? always?) the poly and active layers. A metal layer can be placed on top of a buried contact without creating a connection. Buried contacts can frequently be identified by a light square outline where poly and active intersect, but this is not guaranteed. Sometimes buried contacts must be inferred from context. The takeaway here is: vias and buried contacts form two additional logical layers that need to be vectorized. Buried contacts are different from MOSFET gates b/c a MOSFET gate is not a direct connection. A gate has a layer of insulating oxide separating the polysilicon and the active layer/wells underneath. However, it tends to be obvious from context and visual inspection which type of connection exists, even without having a 3d cutaway view which would show the differences. Not all ICs have the same number of layers or the same layer material type, but in the case of NMOS, it's safe to divide an IC die into at least three layers: Metal, conducting material. Typically a whiteish hue that stands out, in the case of aluminum (which is what YM2151 uses).</li> Polysilicon, pure silicon. Used for MOSFET gates. Color cannot be assumed, but outlines are obvious.</li> Active, doped silicon used for drain and source. Color cannot be assumed, but outlines are obvious.</li> </ul> Let's Digitize!</h2> With the above out of the way, let's digitize the metal layer and vias of the ROM. Please note that in some images that I may miss a section :P. I correct it in a later step unless otherwise noted. Layer Color Table</caption> Layer</td> Color</td></tr></thead> Metal</td> Yellow</td></tr> Polysilicon</td> Red</td></tr> Active</td> Blue</td></tr> Via</td> Green</td></tr> Buried Contact</td> Pink</td></tr> </tbody> </table> Metal layer is easily visible. Here, the ROM matrix, ROM inputs and power and ground rails are digitized. I will show how to infer the latter three shortly.</figcaption> </figure> Here, we have digitized most of the vias of interest.</figcaption> </figure> The image has become a bit crowded after digitizing the metal and via layers, so for the time being I will disable them. Let's start looking for transistors. I personally like to start with pullups, becausepullups have a very distinctive shape on an NMOS die, and the power rail can also be inferred. As mentioned before, NMOS pullups are depletion-mode MOSFETs, and they have very large gate widths to create a current path even without an applied gate voltage. Additionally, to provide the pullup effect, there exists a connection to the active layer on the source side of the MOSFET that looks like a hook. Pullup MOSFETs thus tend to look like "rectangles with a hook", with slightly more emphasis on the hook to create the source-to-gate connection. We can now safely vectorize the pullups, and immediate polysilicon traces emanating from the pullups. We have vectorized our first MOSFET gate! A depletion mode MOSFET at that.</figcaption> </figure> All pullups, polysilicon layer, vectorized. By process of elimination we know the remaining two sides of the ROM are the inputs or outputs.</figcaption> </figure> In our depletion mode pullups, the active layer consisting of source and drain runs through the center of the wide gate. We can trace out the active layer completely now, but I deliberately stopped short. The crossing of two layers near the pullups at the bottom is signficant. We can start vectorizing the active layer that forms the source and drain of the pullups.</figcaption> </figure> Notice that each strip of the metal layer connecting at the top of the ROM terminates in a via. The via connects to a layer below, either active or poly, that runs across the length of the ROM. This layer abruptly terminates after crossing the active layer that directly connects to the pullup's source. There was a transistor formed due to that crossing during fabrication! We can safely assume them to be enhancement mode transistors used as switches due to the gate size. Our unknown layer must be poly because they form the gate of a transistor. Furthermore, Because the metal at the top of the ROM attaches directly to the gate of a transistor for each input, the top of the ROM must be our input. By process of elimination, the output of our ROM is on the right. Our first enhancement mode transistors. Notice how much smaller the gate is for each compared to depletion mode.</figcaption> </figure> Now I decided to take a break from the poly and vectorize the buried contacts. The buried contacts in this section are all visible as squares at poly and active crossings. Since a pullup must have a buried contact to connect the gate and source, let's start with the pullups. Can you find the outline of the other buried contacts before scrolling to the second image? Some buried contacts due to pullup connections have been digitized. All buried contacts of interest in this section are visible.</figcaption> </figure> Remaining buried contacts of interest digitized.</figcaption> </figure> Now, I finish the active layer, which by process of elimination is going to be the remaining unvectorized traces. These form a number of enhancement- mode MOSFET switches distributed through the ROM matrix. Anywhere the poly crosses the active layer is a transitor! Remaining active layer forming the ROM matrix digitized. I accidentally missed part of the active layer at the bottom of the second column when taking these images.</figcaption> </figure> With the active layer (minus my mistake) digitized, the ROM has been fully vectorized. I re-enable the metal and via layers to show the final result. Additionally, I vectorized a few more sections all all layers, only one of which is relevant to the ROM. The thick metal trace below the ROM which connects to the active layer of the ROM matrix (at the source terminals of the enhancement mode MOSFETs immediately attached to depletion mode pullups), is in fact a ground trace. From experience, I can expect the active columns running through the ROM matrix to be connected to ground. I will explore why in the next section. Fully vectorized ROM, plus some extra connections.</figcaption> </figure> Schematic Capture</h2> I am arbitrarily labeling the leftmost and bottommost bus lines the LSBs of the input and output, respectively. Thus, bit positions increase as one travels from left to right and bottom to top of the ROM matrix. Initially, I had intermediate images of my progress creating the schematic. Unfortunately, for various formatting reasons (repeated transistor numbers, inconsistent resolution), the intermediate images didn't turn out how I liked, so I removed them. I created the schematic in a manner very similar to how I vectorized starting with the pullups, then adding the inputs and their corresponding MOSFET switch connections. Then I added the ROM outputs. Next, I added the remaining wires that run down the ROM matrix columnwise, which attach to the source of the switch enhancement mode MOSFETs and pullup depletion-mode MOSFETs respectively. I finished schematic capture by adding the additional switch transistors that exist anywhere the active layer crosses poly within the matrix. For each trio of column wires, the leftmost wire is the ROM input, the middle wire is the active layer running across each row of the ROM matrix, and the rightmost wire is attached to its corresponding pullup. Full schematic. This 5x10 ROM contains nearly 60 transistors!</figcaption> </figure> I made a mistake when drawing the above schematic. By convention, the source should be at a lower voltage than the drain, but for the transistors within the ROM matrix, I accidentally swapped source and drain. In an IC, this does not matter, as a MOSFET is symmetric and swapping source and drain does not affect device operation (for our intents and purposes). However, without this disclaimer, I'm sure I will confuse people. Perhaps the drain and source distinction is best ignored for this schematic. Reading Out the ROM Contents</h2> With the above schematic, we can gleam some interesting information about how the ROM works. I assume the ROM inputs are either always a valid 1 or 0, because I am assuming that this ROM is driven by internal control logic. If any given bus input is 0, the input will not turn on the switch transistors at the bottom of the ROM, placed immediately before pullups. This means that the pullups are not actively driven low, and the source terminal of the pullups remains at a high logic value. The logical high is propogated to all transistors whose gates are connected to the pullup source; these transistors are on. All transistors whose gates are attached to the bus input are in cutoff and have no effect on circuit operation. Notice that the metal corresponding to each bit output is attached to all transistors in a row in parallel. This means that if any of the transistors in a given row are on, the entire metal row, and consequently the output, is pulled low as well. This is also called wired-AND</a>. In a similar manner, if any given bus input is 1, the input will turn on the switch transistor and the logical level at the pullup source will be driven low. All transistors whose gates are attached to the pullup source terminal will be in cutoff and will not drive the metal strips low. However, because the bus input is logical high, any transistors whose gate is attached to the bus input will drive its corresponding bit output low. We now have enough information to devise boolean expressions and a truth table for the entire ROM! Output Bits Driven Low For Each Input Bus Line</caption> Bus Line+Value</td> Output Bits Driven Low</td></tr></thead> I_{0 High</td>} O_{1, O_{6, O_{7</td></tr>}}} I_{0 Low</td>} O_{0, O_{5, O_{8, O_{9</td></tr>}}}} I_{1 High</td>} O_{0, O_{3, O_{6, O_{7</td></tr>}}}} I_{1 Low</td>} O_{1, O_{4, O_{5, O_{8, O_{9</td></tr>}}}}} I_{2 High</td>} O_{2, O_5} I_{2 Low</td>} O_{0, O_{1, O_{3, O_{4, O_{6, O_{7, O_{8, O_{9</td></tr>}}}}}}}} I_{3 High</td>} O_{1, O_{2, O_{3, O_{6, O_{7</td></tr>}}}}} I_{3 Low</td>} O_{0, O_{4, O_{5, O_{8, O_{9</td></tr>}}}}} I_{4 High</td>} O_{6</td></tr>} I_{4 Low</td>} O_{0</td></tr> </tbody> </table> Output Bus Line Equations</caption> Bus Line</td> Boolean Expression</td></tr></thead> O_0</td> ~(~I_{0 | I_{1 | ~I_{2 | ~I_{3 | ~I_{4)</td></tr>}}}}} O_1</td> ~(I_{0 | ~I_{1 | ~I_{2 | I_{3)</td></tr>}}}} O_2</td> ~(I_{2 | I_{3)</td></tr>}} O_3</td> ~(I_{1 | ~I_{2 | I_{3)</td></tr>}}} O_4</td> ~(~I_{1 | ~I_{2 | ~I_{3)</td></tr>}}} O_5</td> ~(~I_{0 | ~I_{1 | I_{2 | ~I_{3)</td></tr>}}}} O_6</td> ~(I_{0 | I_{1 | ~I_{2 | I_{3 | I_{4)</td></tr>}}}}} O_7</td> ~(I_{0 | I_{1 | ~I_{2 | I_{3)</td></tr>}}}} O_8</td> ~(I_{0 | ~I_{1 | ~I_{2 | ~I_{3)</td></tr>}}}} O_9</td> ~(I_{0 | ~I_{1 | ~I_{2 | ~I_{3)</td></tr> </tbody> </table> Extracted ROM Contents of Analyzed Section. Underscores are for clarity.</caption> Input Bus</td> Output Bus</td></tr></thead> 0b0_0000</td> 0b00_0000_0100</td></tr> 0b0_0001</td> 0b00_0000_0100</td></tr> 0b0_0010</td> 0b00_0000_0100</td></tr> 0b0_0011</td> 0b00_0000_0100</td></tr> 0b0_0100</td> 0b00_1100_1000</td></tr> 0b0_0101</td> 0b00_0000_1000</td></tr> 0b0_0110</td> 0b00_0000_0010</td></tr> 0b0_0111</td> 0b00_0000_0000</td></tr> 0b0_1000</td> 0b00_0000_0000</td></tr> 0b0_1001</td> 0b00_0000_0000</td></tr> 0b0_1010</td> 0b00_0000_0000</td></tr> 0b0_1011</td> 0b00_0010_0000</td></tr> 0b0_1100</td> 0b00_0000_0000</td></tr> 0b0_1101</td> 0b00_0000_0000</td></tr> 0b0_1110</td> 0b11_0001_0000</td></tr> 0b0_1111</td> 0b00_0001_0000</td></tr> 0b1_0000</td> 0b00_0000_0100</td></tr> 0b1_0001</td> 0b00_0000_0100</td></tr> 0b1_0010</td> 0b00_0000_0100</td></tr> 0b1_0011</td> 0b00_0000_0100</td></tr> 0b1_0100</td> 0b00_1000_1000</td></tr> 0b1_0101</td> 0b00_0000_1000</td></tr> 0b1_0110</td> 0b00_0000_0010</td></tr> 0b1_0111</td> 0b00_0000_0000</td></tr> 0b1_1000</td> 0b00_0000_0000</td></tr> 0b1_1001</td> 0b00_0000_0000</td></tr> 0b1_1010</td> 0b00_0000_0000</td></tr> 0b1_1011</td> 0b00_0010_0000</td></tr> 0b1_1100</td> 0b00_0000_0000</td></tr> 0b1_1101</td> 0b00_0000_0001</td></tr> 0b1_1110</td> 0b11_0001_0000</td></tr> 0b1_1111</td> 0b00_0001_0000</td></tr> </tbody> </table> As we can see, a number of inputs result in zero state outputs, and the MSB only changes the output of two ROM entries depending on whether its set or not. Perhaps a number of these states are illegal and just given a default output? I wonder what this ROM is used for? When I figure it out, I'll make an edit to this page! Future Direction</h2> As readers can probably see by now, digitizing and REing old ICs is completely doable, if tedious. Personally, I would say it's more mechanical than reversing a binary with IDA or radare2, once you know what to look for. However, like REing a binary, it does take a long time to fully RE an IC. There are tools to automate the schematic capture process of an IC, and aid in analysis as well. Olivier Galibert's dietools</a> are one example that I hope to discuss in future posts. As of writing this post (October 12, 2016), my work in vectorizing and schematic capture of the YM2151 can be found here</a>. Anecdote</h3> Back in 2011, I discovered that the MAME project was decapping ICs to defeat security/protection circuits on old arcade boards that prevented them from being emulated properly. The me in 2011 thought this was the most fascinating thing, the "last bastion" of proper accurate emulation. I never thought I would have the skill set required to do IC analysis. Even up until summer 2016, I said that I wouldn't do IC reverse engineering, despite preservation of old technology being important to me. I felt it was beyond my comprehension, and that I would not be able to learn how to identify features in a reasonable amount of time. With help from others, I was wrong, and I'm glad that I was. If you're on the fence about learning a new technical subject, don't hesitate. We're all smart, and filled with doubt. Others will be willing to help! Thanks</h3> I would like to thank members of siliconpr0n for looking over this post, especially Olivier Galibert for correcting a few mistakes. Additionally, I'd like to thank Digi-Key for their extremely useful Scheme-it</a> schematic program, which I used to create the schematics (including the nice arrow!). Floppy Disk Notes 2016-02-10T00:00:00+00:00 Floppy Disk Notes</h1> What follows is a set of links that I've collected over the years that give technical information on how to encode/decode floppy disks signals, as well as the theory of operation behind this dying medium. I find reading these documents important for preservation purposes, especially since computer programmers of the past relied on these engineering details for copy protection purposes. Additionally, a deep understanding of internals may one day help to recover data that is thought to be lost using statistical analysis of the read signal. Links are ordered relatively in the order I read them/I recommend reading them, and sections tend to build upon each other. General</h2> The Floppy User Guide</a></dt> A good technical overall technical description of how a floppy drive accesses data.</dd> </dl> Floppy Drive</h2> SA800/801 Diskette Storage Drive Theory of Operations</a></dt> Without question, the most important document on this list. If you read any document, read this. It's not quite enough information to build a floppy drive from scratch, but it's enough to bring someone interested up to speed. Hard to believe this document is 40 years old in 2016!</dd> SA850/SA450 Read Channel Analysis Internal Memo</a></dt> This internal memo donated by a Shugart employee includes a floppy drive read head transfer function analysis based on experiments Shugart did in the late 70's.</dd> </dl> Phase-Locked Loops (PLLs)</h2> Phaselock Techniques, Floyd M. Gardner</a></dt> A monograph on analog PLLs. Does not discuss All-Digital PLLs (ADPLLs).</dd> NXP Phase Locked Loops Design Fundamentals Application Note</a></dt> A quick reference for analog PLL design.</dd> </dl> Encodings</h2> MFM</abbr></h3> Floppy Disk Data Separator Design Guide for the DP8473</a></dt> To be written.</dd> Encoding/Decoding Techniques Double Floppy Disc Capacity</a></dt> Gives background on more complicated physical phenomenon associated with floppy drive recording, such as magnetic domain shifting.</dd> Floppy Data Extractor</a></dt> A schematic for a minimum component data separator that does not require a PLL, but uses a digital equivalent. Perhaps a simple ADPLL</abbr>?</dd> </dl> RLL</abbr></h3> IBM's Patent for (1,8)/(2,7) RLL</a></dt> I'm not aware of any floppy formats that use (2,7) RLL, but hard drives that descend from MFM floppy drive encodings do use RLL. RLL decoding is far more involved than FM/MFM.</dd> </dl> GCR</abbr></h3> This is a format used by Apple II drives and descendants. Software has more control over this format, so there are more opportunities for elaborate data protection compared to the IBM platforms. TODO when I have time to examine non-IBM formats. Track Formats</h2> IBM 3740 (FM, Single Density)</h3> TODO. Described in Shugart's Theory of Operations manual. IBM System 34 (MFM, Double Density)</h3> TODO. Described in various documents on this page, but I've not yet found a document dedicated to explaining the format. Floppy Disk Controller ICs</h2> NEC 765</h3> 765 Datasheet</a></dt> The FDC used in IBM PCs. It is not capable of writing raw data at the level of the IBM track formats. Thus, attempting to write copy-protected floppies is likely to fail with this controller.</dd> 765 Application Note</a></dt> NEC created an application note to discuss how to integrate the 765 into a "new" system, either using DMA or polling on receipt of interrupts.</dd> </dl> TI TMS279X</h3></h3> TMS279X Datasheet</a></dt> Includes a diagram of the IBM System 34 track format.</dd> </dl> NI DP8473</h3> DP8473 Datasheet</a></dt> A successor to the 765 that is capable of handing formats such as 1.2MB High Density (HD) disks</dd> Design Guide for DP8473 in a PC-AT</a></dt> TODO. It appears I lost my original commentary on this document.</dd> </dl> Floppy Disk Controller Cards</h2> IBM PC FDC Card (765)</a></dt> Includes schematics. The PLL circuit on the last page is in particular worth analyzing.</dd> </dl> If anyone has any interesting new documents to add, please feel free to contact me, and I will add them to this page with credit!}}}}}

WDJ

How To Download Your Twitter Archive Using `curl`

Preparing The Browser And Monitor</h3> Both Firefox and Chromium-based browsers provide tools to get the required data- the Network Monitor. This post focuses on Firefox because that's what I use, but the process for Chromium should be similar.</p>

Using `z3` To Solve Logic Puzzles

A Tour Of ethflop

Usage</h2> Starting and Stopping ethflopd</code></h3> To start the daemon, supply the following command line arguments in order:</p> The network interface you wish to listen on (eth0</code>, wlan0</code>- wireless interfaces work</em>, etc.) .</li>

Starting and Stopping ethflopd</code></h3> To start the daemon, supply the following command line arguments in order:</p> The network interface you wish to listen on (eth0</code>, wlan0</code>- wireless interfaces work</em>, etc.) .</li>

Loading/Unloading the TSR</h4>

Packet Driver Installation</h5> The first</del> second</ins> thing to do is to make sure your packet driver is loaded into memory. In my case with the 3c509, this looks like:</p>

Querying commands</h4> You can get information about the ethflop TSR and what disks are available by running ethflop s</code> and ethflop l</code> respectively:</p>

Using Virtual Disks</h4> Commands to operate on virtual disks are the bulk of ETHFLOP.COM</code>'s functionality. Any typical action you would perform on a physical floppy disk has an equivalent coded into ETHFLOP.COM</code>:</p>

Unsupported Hardware Floppy Density Emulation</h6> You can even create images of floppy densities that the hardware doesn't support! In the below snippet, I create a 1.44MB 3.5" virtual disk and populate it much the same way as the previous examples, minus the quotes this time :):</p>

Porting a New Board To Migen

Porting a New Board To Migen</h1> Taps mic</em> Is this thing on? So it's really been 11 months since I last wrote something? I really need to change that! I'm going to attempt</em> to write smaller articles in between my larger, more involved ones to keep me motivated to keep writing.</p>

Leveraging Python To Build FPGA Applications</h2> To motivate how to port a new development board, I need to show Migen code right now. If you haven't seen Migen before now, don't panic! I'll briefly explain each section:</p>

NMOS IC Reverse Engineering

Floppy Disk Notes

Encodings</h2>

Track Formats</h2>

IBM 3740 (FM, Single Density)</h3> TODO. Described in Shugart's Theory of Operations manual.</p>

IBM System 34 (MFM, Double Density)</h3> TODO. Described in various documents on this page, but I've not yet found a document dedicated to explaining the format.</p>

Floppy Disk Controller ICs</h2>

IBM 3740 (FM, Single Density)</h3>
TODO. Described in Shugart's Theory of Operations manual.</p>

IBM System 34 (MFM, Double Density)</h3>
TODO. Described in various documents on this page, but I've not yet found a document dedicated to explaining the format.</p>