powered by

FTD

How To

FTD Journal

Conditional Variable

Sublime Text Gets Syntax Support

Index Symbols

10th Feb 2022

Now that we are creating a lot of FASTN packages, question of documentation and best practices starting to dictate our features. One thing I have struggled with is how to organize a package.

Till a while back the only ergonomic way was to put everything into one single index.ftd file. We had to do single file else there would be a lot of import statements in each file. Also it interrupts the writing, if you are in the middle of a ftd file and decide to use a new component, you have to temporarily jump to the top of the file, add an import statement and then come back to your original place and use the component.

Now that we have auto import feature, one can easily import more than one module from a package in every file. One would ask people to add a bunch of auto imports when adding a dependency, and we are done.

This means we can now breakup our package into smaller files. So the question is how small a file?

When writing documentation I started with putting the entire documentation in the index file. But then the page is becoming quite long, so I felt I should split the documentation file as well.

I think it would be better if we created a file for each component, and in that file we put the documentation as well as the component source. But when we do the ideal name for the file would be the name of the component itself. Also since ftd only supports importing a module and not individual symbols from the module we would have to end up writing:

-- import: some-lib.com/foo/header
-- header.header: something

I am not liking the header.header etc. What if we allow people to write:

-- import: some-lib.com/foo/header
-- header: something

And ftd will see if we are trying to use the module itself, and if so it will translate it to module.<whatever is named after module>. It can be either a variable, record or component.

The only main downside is if we see something like above we do not know if header refers to some imported symbol, in which case I would go review auto-imports in FASTN.ftd file, or a component defined locally in current file. Since author files usually won’t have components defined or may be one component defined, in general we do not have this issue.

As long as we support any feature like alias, we will have to review FASTN.ftd to see the symbol’s full name. The only way to avoid jumping to FASTN.ftd to understand each component used, is to not use auto import or dependency alias feature, and put explicit import on top of each file.

Final Decision On References

2nd Feb 2022

Let’s hope it is really final, Arpita is now going to implement it.

Every binding is now by reference. So if you define:

-- string foo: hello
-- string bar: $foo

Both foo and bar have the same storage in memory and both are names refering to same thing.

One can opt out of this by cloning:

-- string foo: hello
-- string bar: ftd.clone: $foo

We will build ftd.clone in due time.

Any attempt to modify anything would be governed by the mutable-by rules. The exact syntax of mutable-by is not fixed but concept is.

We will further allow mutable-by to following package interfect. So e.g.

fastn.ftd

-- boolean dark-mode: false
mutable-by: fastn.dev/dark-mode-switcher

Now any package that implements fastn.dev/dark-mode-switcher will be able to modify the fastn.dark-mode variable.

Further anyone that uses such a package will have to still opt in:

amitu.com/FASTN.ftd

-- fastn.dependency: arpita.com/dms
implements: fastn.dev/dark-mode-switcher

Here amitu.com/FASTN.ftd has explicitly granted arpita.com/dms fastn.dev/dark-mode-switcher “rights”.

Reference Vs Value Recap

1st Feb 2022

So Arpita is working on -- boolean foo: $bar and we are stuck at the question we discussed on 29th Jan.

-- boolean y: true
-- boolean x: $y

-- foo:
boolean $x: $y
boolean z: $y

In foo we are going with the proposal Arpita likes for declaring variable references: boolean $x meaning $x is now tied to the rvalue. If rvalue changes $x will change. z is assigned rvalue at declaratin and does not change afterwards.

Crux of the problem: why do we not always have references? If we do any local change in component will be reflected globally if any variable was to default to a global value. Many times we want defaults set by global, but if inadvertantly every component starts modifying globals it would be a mess.

The specific problem Arpita had in above snippet was now we have different syntax, if global assignments are always reference, but in component its always value then we have an inconsistency which she rightly doesn’t like.

One option is we go with her syntax, <type> $x means its an alias, <type> x means it is a assign current value. My syntax was <type> variable x vs <type> x. If we go with either of them, does it solve our problem?

The thing is component changing global variable should be exception, but component getting latest value of global should be norm.

Question: why are we then creaitng a new variable in component, and not just using the global variable from inside the component? We want to use global variable as the default value of some component property, but we want callers of component to change that if they want to.

Would readable vs writable semantics help? If we are doing:

-- ftd.row foo:
boolean x: $some-global

We want to default x to some-global, but want callers of foo to be able to overwrite it. We are not creating a variable to change it in the component after compnent has been constructed.

Currently there is no way to declare if a variable is mutable or not. If there was we could have different behaviours: a mutable variable will not reflect changes to global, or maybe a mutable variable will not reflect changes to global after it has been mutated once. But it will never mutate global.

Should we say its always reference and always non mutable. To make it mutable you opt-in, and then behaviour changes. Or we decide if it’s mutable by analysing the body of the component.

Say we use a explicit mutable syntax, and only allow event handlers to modify a mutable variable:

-- ftd.row foo:
mutable boolean x: $some-global

Or we do:

-- ftd.row foo:
boolean $x: $some-global

But sometimes we want components to modify top level variables as well.

How about we go with this bevaviour: we don’t have mutable, any component variable with default value of global will keep reflecting changes in global till it has been modified by the component, once it is modified it starts ignoring global changes. Further, a component can modify global by directly using the global variable in the event handlers.

We will still need reference sytnax: the above the was the default behaviour of normal variables. We will still have <type> $foo or <type> variable foo to indicate it takes a variable from caller context and it always modifies that variable.

How about global reference:

-- boolean y: true
-- boolean x: $y

-- foo:
$on-click$: toggle $x

Does it modify only global x or y as well? We can say we follow the same behaviour for global variables as well, x will continue to track y till x was modified. Unless x was defined as boolean variable x, in which x and y are always the same variables.

Final Proposal?

All variables, where-ever they are defined, if they are defined with rvalue of an expression involving one or more variables will continue to reflect the value of expression at that point of time, till the variable has been explicitly overwritten.

Identity expression on a variable $x is simply $x, any other expression, eg $x * 2 is non-identity (even $x * 1 or $x + 0 are non identity expressions).

If a variable is defined with non identity expression on another variable it is considered a formula and can not be changed. Non identity expressions can not be used as rvalue of <type> variable x.

Objection: Variable changes meaning on runtime. For a while it will follow another variable and suddenly it will stop. It would be hard to debug. We can put a warning on console everytime it happens, when a variable changes “nature”, we can even give warning on build time that there are now variables that change their nature.

Microsoft PowerFX Model

One language/system to compare FTD with is Micosoft Power FX. They have an interesting aspect, they do not have “global data”, all data is in some UI element, and only that UI can change that data. So you get this garurantee:

To understand where Label1’s text is coming from, you know exactly where to look: the formula in the Text property.

This is huge in terms of debuggability of a complex Excel/PowerFX codebase. Everything is pure. There is no concept of time. Only internals of a component can modify a variable, from outside a component you can not.

The problem with them is the same, everything is wrapped in UI components. FTD has a pure data language representation, we are trying to keep data in globals so they can be easily accessed from programs, and components modify these globals, but since any component can modify any global, we have a potential debugging nightmare.

So data only in UI, no globals, and pure expressions over these ui specifi data? Or data first, and UI later to modify the data? In Excel/PowerFX it’s a bit hard to have two “views” that can modify same underlying data. Consider Dark Mode, we model it as a global, fastn.dark-mode, but in PowerFX it would be managed by a cell, and what if you want to have dark mode switcher in multiple places in the UI? Since they are different components either it would not be possible, or PowerFX will have an escape hatch and will make some sort of singleton variable shared by all instances of the given component. We can do this as well, create “static” or singleton modifier for variables, and share it across all instances of a component.

But then we are still facing all data is in UI problem. We have to identify UI elements using IDs, or name of component if they are singlton. Current clean data modelling will go.

Limited / Controllable Mutability

It’s not easy to put all data only in component, data being defined at top of file, etc is handy, we have seen it work quite well, everything looks clean etc, but we are struggling with components modifying gloabls problem.

What is we say globals of a module or a package can only be modified by components defined in that module or package? We still allow top level modifications, eg config modifies colors etc, this is not part of component but top level declarations.

If we had way to limit modifications, eg mut(package) boolean foo: we are saying only current package can modify this foo. Default can be mut(module) meaning only current module can modify it. And we also allow mut meaning anyone can modify things. Or we change defaults, mut means mut(package), and mut(pub) or something to mean everyone can modify.

So we go with the “Final Proposal” with mut added to limit debugging scope?

-- boolean foo: true
mutable-by: package
mutable-by: module
mutable-by: all
mutable-by: some-other-package

Maybe we can even allow name of other packages explicitly that can mutate it. mutable-by can be specified multiple times. package and module refer to current package and module respectively, or you can specify any package or specific module. Or you can specify all to say everyone can mutate this. By default it would be mutable-by: module.

Boolean Expressions

29th Jan 2022

We are going to add boolean expressions in FTD now. Till now we support if at component level, attribute level, and variable level.

-- foo:
if: $something
color if $something-else: red

-- string bar: yo

-- bar: hello
if: $something

In the first example we use if both at component level, and attribute level. In the next example we use if to control bar variable.

Lets call if: as section level if, and foo if $something as attribute level if.

Till now we can either do $something or not $something, but not higher boolean expressions like if: $a or $b etc.

We want to allow general purspose boolean expressions now, with and, or, and grouping using (/). We will follow Python syntax for that.

What about multiline expressions? We can use multiline expressions for section level if easily:

-- foo:
if: (
    $a or $b or (
        $something and $something-else
    )
)

-- foo:
if: $a or $b or (
    $something and $something-else
)

Should we also allow:

-- foo:
color if (
    $a or $b or (
        $something and $something-else
    )
): red

Technically we should. We will have to fix our p1 grammar/parser for this. We can always implement the inline versions first, and do the more general syntax later on.

Other thought was we created intermediary “formulas” and use them.

-- boolean some-formula: (
    $a or $b or (
        $something and $something-else
    )
)

-- foo:
color if $some-formula: red

We should also allow component level formulas when defining a component:

-- ftd.row foo:
boolean some-formula: (
    $a or $b or (
        $something and $something-else
    )
)

\--- ftd.text:
color if $some-formula: red

Basically we can chose to only support expression at header or caption level, and not a p1 key level. In fact maybe not even at cation level:

-- boolean some-formula:

$a or $b or (
    $something and $something-else
)

Above is arguably cleaner than the caption one. Which leaves us with header being the only thing that needs multi value support. Unless we allow this syntax:

-- ftd.row foo:

\--- boolean some-formula:

$a or $b or (
    $something and $something-else
)

Here we have created a “component level variable” by using ---. With component level variable we never need multi line header key or value to support multiline boolean expressions. Which means we won’t have to modify our p1 parser and grammar remains simpler.

Arpita thinks even if we use multiline value or caption, we must use our continuation syntax:

-- ftd.row foo:
boolean some-formula:
> $a or $b or (
>     $something and $something-else
> )

With continuation in the mix we do not need “component level variable”. Or we support both continuation and component level variable.

Pass By Value Vs Reference

If we do this:

-- boolean foo: true

-- t:
val: $foo

-- ftd.text t: hello
boolean val:
color if $val: red

If we modify $foo, should $val change? This is currently implemented as pass by value. We are planning a syntax in future:

-- t:
val: $foo

-- ftd.text t: hello
boolean variable val:
color if $val: red

Here we defined val as boolean variable instead of boolean as in last example. With x variable, the we will implement pass by reference semantics. We call this feature higher order variable.

Arpita proposes this syntax:

-- t:
val: $foo

-- ftd.text t: hello
boolean $val:
color if $val: red

It’s better because it’s way shorter.

One open question for higher order variable is should this be allowed:

-- t:
val: true

-- ftd.text t: hello
boolean variable val:
boolean $val:
color if $val: red

Here we have passed true but $val need to bind to some existing variable, and we only have a value. We can say we would allow it, and any modifications to $val would be lost, semantically its modifying that true, but since nothing else is bound to it the modification is not visible outside.

With this we do not have any major objections to higher order variable.

So we have two possible variables, changes in parent reflect in child and child in parent when we use higher order variable. Or change in neither side is reflected in other side, when using normal variable (pass by value). The default of pass by value seems bad, as if a parent variable is modified it’s value is updated everywhere it is directly referenced but not in children:

-- ftd.row foo:
boolean o: true

\--- bar:
o: $o
color if $o: red
$on-click$: toggle $o

Here color keeps changing when $o is changing, but o is not. This feels wrong.

Shouldn’t people just write:

-- ftd.row foo:
boolean o: true

\--- bar:
o if $o: true
o if not $o: false

It clearly looks equivalent to above but is actually different.

I am not saying pass by reference should be default, else any changes done by bar to $o will always be visible to parent. I mean we can do that. But I feel there should be a third choice where changes go from parent to child always. If the child component is modifying the variable that modifications is not reflected outside, but if not then it is.

If the variable was never modified by a component then the value passed from outside could always be reflected and there is no concern. But if component is updating a variable then what to do? We can say if variable is never modified by component then we do pass by reference, else we do pass by reference or value depending on if component has defined variable as higher order or not.

Arpita says if we do this it would be hard to explain as the behaviour of a component will suddenly change if they add an event handler that modifies a variable.

What to do?

MOUSE-IN Behavior Change

Currently we have a special variable $MOUSE-IN which can be used anywhere inside a component definition without declaring, and it is true if the mouse is hovering over the current component.

This is limiting, what if you wanted to use this variable from outside a component? Or what if you want to be more precise, within a component if you want to do something if the mouse is within a specific child in the component?

If you want to emulate the old behaviour you will have to do this:

-- ftd.row foo:
boolean mouse-in: false

\--- ftd.text: yo
$on-mouse$: $mouse-in=$MOUSE-IN

We have $on-mouse$ that will be triggered when mouse enters or leaves a component. Then we can do the following as well:

-- ftd.row:
boolean mouse-in: false
border-width if $mouse-in: 1

\--- ftd.text: yo
$on-mouse$: $mouse-in=$MOUSE-IN

Here we are not defining a component and can still use $MOUSE-IN.

Comics Using FTD

27th Jan 2022

We can already create comics using FTD. But we need is more succinct syntax to represent a bunch of position, scale, rotation related attributes into a single short-hand notation, and eventually give a drag and drop interface that updates the position etc shorthand notation.

-- panel:
d: 0, 0, 1

\--- girl-1:
pose: flying
d: 0, 100, 1.5
a: left-right

\--- girl-2:
pose: flying
a: bottom-top, 0-1, left-right, 1-2

\--- girl-2: Hello Everyone! We are Power puff girls
pose: flying
a: 2-

\--- girl-3:
pose: flying
a: right-left, 1s

d: could be dimensions, position, orientation, scale etc. a: could be animation. We can even have double representation, eg a separate place where all the dimensions and animations are put for the entire scene, so rest of the code only shows the character names, poses, dialogs etc.

If we do this, we can make the “source” of comic almost as readable as the rendered comic itself. And this technique is not limited to only comics, any thing that requires spatial arrangement, but is can otherwise be described better in text, say an diagram, model of a house and so on can all benefit from such a representation.

We tend to think that such things can not be represented in text, but maybe it’s only because the optimal representation has not been created yet.

The advantage of text is tremendous. We can use Git etc, the only technology worthy to be called collaborative, Google Docs, whose entire reason for existence is “word processor collaboration”, can not scale to dozen collaborators, forget 100s or tens of thousands, which git can easily. None of the Figma’s, Adobe’s offerings, random comic generation tools, equation editor, animation creator, etc work with reasonable number of collaborators, who would create an account with every diagramming, comic generation, image editing tool you use. There is just too much fragmentation, entire companies on individual tools, but your workflow needs all of them. Git can solve that if there was a text file representation for each need.

Pure Functions?

26th Jan 2022

Was discussing function definition stuff with Arpita and I am quite happy with the design in general. We have can pass variables to functions by reference (so they can be modified by the function), or value. Function decides if the passed argument is by reference or value. Beyond these referenced variables, a function can not modify any variable. Function can read other global variables but can’t modify them.

I also like the either a function call is used as a. event handler, in which case function must have a return type of void or null, or maybe we can even call them event-handler to make the distinction clearer, or b. formula, a function whose output is “bound” to a name, every time any of the arguments of the formula, or any global reference in the formula is changed, the value of the bound name is also recomputed and changed.

Error handling with result and option being special data types, and our if expression handling them is also still making sense after a few days of deliberations.

What about async? What if a formula needs to depend on say local store, or some API? Should we support it? I do not yet have a good answer to this. I feel like saying just switch to “host language” for such things, where you get full first class language, instead of FTD. But do I want to say FTD is not a first class language? Where to draw the line? We can obviously not give all the host capabilities to ftd, I mean we discussed reading from API, but what about reading from USB or file system or Windows Registry, etc etc? We are going to have to depend on host functions to give access to such features.

We can not be a single language. We are UI language. Data manipulation and arbitrary computation (within limits) on the data should be supported first class, but we can not go for complete replacement. We also have to live side by side with another “host” language.

But is host language “user programmable”? Or is it only for giving users building blocks, that they can then connect together? This drawing a line between what must forever remain host language domain, and what must be handled by ftd is tricky to resolve for me yet.

Dynamic Components: Path Ahead

21st Jan 2022

Now that we have basic dynamic components working, we need to think of syntax for more actions. We currently has “SQL-esque” commands, eg increment $foo by 1 etc, instead of function like syntax: $foo.increment(by: 1). We have to figure out what we want our eventual syntax to look like.

Some command operations:

  1. increment an integer variable: increment $foo by $incr clamp 10.
  2. Updating a variable: $foo=<new value>.
  3. Toggling a boolean: toggle $foo
  4. Inserting an element into a list: insert $el into $list at end or at front or at <position>.
  5. Clearing a list or an optional value: clear $list or clear $optional
  6. Removing element at given position: remove from $list at $pos
-- ftd.text: hello
$on-click$: increment: what=$count, by=2
> what: $count
> by: 2
> clamp: 10

-- ftd.text: hello
$on-click$: increment: what=$count, by=2, clamp=10

Moving to a function call syntax looks good. We have to prefix functions by their module names, so most of functions provided by ftd will be ftd.<foo>. Arguments will always be named, no positional arguments. We will use = for arguments that come on same line, and : for arguments that come in subsequent lines. Functions can define caption type, in such cases, other arguments can not come in caption line.

Function Syntax Basics

One possible way to implement functions could be:

-- void increment:
integer variable what:
integer by: 1
optional integer clamp:

integer new = $what + $by

if $clamp && $new > $clamp {
    $new = 0
}

$what = $new

The type of a function is in the declaration line. We have created a special type void for functions that are only for side effects, e.g. to be directly called from $on-*$. For other types, eg integer it will appear to look like an integer variable definition, but since it takes arguments, it’s a function.

The last expression can be considered the return type of a function, but we will also support return <value>. Arguments are specified in usual manner.

Higher Order Variables

In the example we have used integer variable what, which is to indicate that here we are not accepting a value, but the reference to some variable, which would be modified by this function.

Functions Vs Formula

Functions would be called from event handlers. Functions job is to mutate some variables.

Some functions can be defined as pure, they can not take higher order variable, and can only be called from variables defined at global levels. Formulas can depend on other variables, and when the underlying variable changes, the value of the formula changes automatically.

-- integer length: 20

-- integer foo: area: length=$length, width=20

Here since the declaration of a variable uses a function, foo is considered a formula. In such cases further updates to foo is not allowed. If $length changes, foo would be auto-updated.

Error Handling

We are basically stuck here right now. What is you try to remove the 10th item from a list, but the list only contains 2 items? Should it silently be dropped? We do type checks, so we do not have to worry all sorts of errors, so you can not add a string to an integer array, this will be checked at “compile time”, if you try to do that, the document would fail to load. But run time errors can still happen.

Variables change from events. What kind of errors: division by zero. Parsing error, e.g. if someone is trying to enter an integer in an input field, and they type some non digit. Or say if we are reading some data from JSON, and we are expecting an integer but we get a string, or the key is missing.

Among these errors I listed, divide by zero is special, in being an operation that we usually don’t want to make “error check before proceeding”, for others we can ensure all functions that can fail return some value that forces one to handle failure possibility. Forcing error check for divide by zero will force all mathy code to become too verbose.

Any error that happened during initial page load is “fine”, we simply bail, show error message and let author figure it out. This is equivalent to “compile time”, and we want people to not store invalid documents. But runtime errors we can not avoid.

Rust Like Result And Option

Rust has first class for failures, Result and Option types are there. It kind of means we need some level of generic support. We can have special handling for Result and Option types without generic.

So we can write something like:

-- integer result to-integer:
string value:

Here we have a function to-return that returns integer result, (for list it would be integer list result foo:).

Handling Errors

if <var-name> = <result expression> { success clause, can use <var-name> } else <error-name> { error clause }

-- void foo:

let i = if num = to-string: value=hello {
    num
} else error {
    console.log(error)
}

Can also work with optional values. Function can return -- optional integer foo: optional values as well. In such cases, the else block won’t take error variable name, it will always be null.

Error type will only be string. In future when we have enums, we can also do:

-- string enum Errors:
invalid-integer: Value Is Not A Valid Integer
empty-value: The value was empty string
overflow: The value is too large to fit into integer type.


-- void foo:

let i = if num = to-string: value=hello {
    num
} else Errors.invalid-integer {
    console.log(Errors.invalid-integer.message)
} else Errors.empty-value {
    console.log(Errors.empty-value.message)
} else error {
    console.log(error)
}

Here we have multiple else clauses for enum.

<type> result is not an or-type

or-type is our name for enum in Rust.

In future we will have match statement for handling or-type, but we are not going to treat result and optional as or-type, so match can never work with them.

Returning Errors

We will use return 1 to indicate success, and fail "whatever the message to return failure message. We will also support let num = to-string? value=hello, here ? acts like Rust.

Event Handler Errors

What happens if one of the event handlers raises an error? One option is we do not allow functions that can fail as event handlers.

Old Notes

insert syntax, clear, delete at index, is list empty, how many items in the list, similarly how to filter

default record

-- void do-get:

fetch-people(after-get, on-error)

-- void fetch-people:
success(person list) -> void:
failure error->void:

ftd.http-get("https://foo.com", success, failure)

-- void after-get:
person list people:

-- void on-error:
error error:

Dynamic Components Landed!

19th Jan 2022

Till now when a page used to get rendered from backend, it used to contain the entire component hierarchy, and front-end event handling etc only changed some of the DOM attributes, and visibility. If there was an element whose visibility was affected by an if clause, it used to be always included, and set as hidden and when needed we used to make it visible. We never created DOM node till now as part of event handling. This means it was not possible for us to have say a list, and add elements to the list from any of our event handling code.

Yesterday Arpita implemented support for dynamically constructing DOM when say we add elements to a list.

–Consider this:

-- ftd.column:
spacing: 10

-- ftd.row:
width: fill
spacing: 10

-- ftd.input:
placeholder: Type Something Here...
width: 250
border-width: 2
padding.px: 8
$on-input$: $query=$VALUE

-- ftd.text: Add
if: $query is not null
color: $inherited.colors.text
background.solid: $inherited.colors.background.base
move-down: 4
padding.px: 5
$on-click$: insert into $strings value $query at end
$on-click$: clear $query

-- ftd.text: Clear
color: $inherited.colors.text
background.solid: $inherited.colors.background.base
move-down: 4
padding.px: 5
$on-click$: clear $strings


-- ftd.text:
if: { $query != NULL }
color if { $ftd.dark-mode }: $inherited.colors.text

You have typed: {value}

-- ftd.text value: $query
color: $inherited.colors.success.text
color if $ftd.dark-mode: $inherited.colors.text
background-color if $ftd.dark-mode: $inherited.colors.background.base
padding if $query is not null: 5
role: $fastn.type.copy-large


-- show-data: $obj
$loop$: $strings as $obj

We are very proud of this feature! What is even cooler is how little work it took to implement it. Initially I thought we will have to somehow send our component spec to front-end, for the components that are directly or indirectly called from the components used by such dynamic elements we want to create. And it sounded like a lot of duplication of logic between what we do in Rust and we would have to do in JavaScript.

Arpita got this brilliant insight that we can create one instance of each such component and keep them hidden, and when we have to create any new instance of any of these components we clone that DOM node, and set it up with correct data.

It turned out to be as simple it sounds, and entire insight to merge was 3-4hrs! I was worried about JS size increase etc, and none of that happened. So little code, such a massive impact.

Now that we have support for creating nodes dynamically, we are almost complete UI solution, and FTD should be considered an alternative anywhere you would consider using say React for creating UI.

“$main$” Variables

18th Jan 2022

Some variables, like fastn.document-title, could be set by more than one ftd documents, each document is telling us what is the document-title for that document by setting fastn.document-title.

But we allow authors to import any document from any other document, barring cycles, so these variables can get clobbered. Say if there is a.ftd and b.ftd and former wants document title to A and later to B, they can do both: -- fastn.document-title: A and so on. But what if a.ftd looks like this:

-- import: fastn

-- fastn.document-title: A

-- import: b

b.ftd would be imported after a.ftd has updated fastn.document-title, and it will overwrite fastn.document-title. We can argue this is fine at some level and user is aware of things.

Further there is another concern, what if fastn.document-title was optional, and the idea was if this is not set, we pick the title from first heading of the document. And say this worked out fine, so a.ftd will not want to fastn.document-title, but say it does not work for b.ftd so it wants to explicitly set it. But then a.ftd imports b.ftd. We can not solve by import order etc.

Proposal:

fastn.ftd file

-- optional string document-title:
$main$: true

If any variable or list is defined as $main$, then only the changes done by the “main document” would be kept. FTD interpreter knows whats the main document that is being interpreted, and it knows when it is interpreting one of its dependencies (via import).

We can further allow $main$ during variable change also to allow non main documents to update variable still, eg a config.ftd or some file which is only for setting project variables.

config.ftd

-- import: fastn
-- fastn.site-title: AmitU's Blog
$main$: true

Further when any “main” document is importing some document, it can declare that that document be also considered main:

a.ftd

-- import: ftd-dev.vercel.app/config
$main$: true
-- import: b

Here we have imported config as “main”, but not b.

This would be applicable for both variable overwrite, and for adding a an element to a list, either can use “$main$”.

In Other Words

  1. some variables can be declared as $main$.
  2. a $main$ variable can only be modified by
    1. document that is considered $main$, 3. and 4. decide when a document is considered $main$
    2. or by forcing update by include $main$ while updating the variable
  3. the document that started the interpreter is the initial $main$ document
  4. when a $main$ document is importing some document it can declare those document as $main$ as well

Edge Case

Consider if b.ftd:

-- import: fastn
-- fastn.document-title: B

-- ftd.text show-title: $fastn.title

And say a.ftd was:

-- import: fastn
-- fastn.document-title: A
-- import: b
-- b.show-title:

The show-title will show A and not B even though if you look at source of show-title in b.ftd it appears it is modifying a variable and using it right away.

This is not particularly confusing because a variable could have been updated multiple times, even within the same file:

-- import: fastn
-- fastn.document-title: B

-- ftd.text show-title: $fastn.title

-- fastn.document-title: New B

In this case, the immediate value of fastn.document-title is not used by show-title, but the “eventual value”.

Object Gets $ref syntax

Arpita implemented variable referencing in object constructor syntax, so now you can do:

-- import: bar
-- object foo:
key: $bar.baz

And say if bar.baz is 20 (integer), it will generate this:

{
    "key": 20
}

Proposal: message-host to only accepts objects

This brings use to message-host. It currently accepts either a string, which is the name of function exposed by the FTD host, or it can be an object, that has to define a key function, which will be the function.

In general I am not fond of “string values”, because if you make a typo, ftd interpreter can’t help you. The host functions are provided by host language, and ftd compiler has no idea about host functions. Ideally FTD compiler should have some knowledge about capabilities provided by host, maybe some sort of schema and type definition?

In the meanwhile, accepting both string and object, I now consider an anti-pattern, we should only support well known object constructors:

-- import: fastn

-- ftd.text: Switch To Dark Mode
$on-click$: message-host $fastn.switch-to-dark-mode

Is superior to message-host switch-to-dark-mode, where switch-to-dark-mode was a string, but $fastn.switch-to-dark-mode is a reference to something FTD interpreter can verify.

$processor$, async, Library::get() and Document::set_*()

16th Jan 2022

One of the things that has emerged since yesterdays brain storm with Arpita is we need first class support for updating FTD variables from “host languages”.

You can say we have already two host languages, Rust and JavaScript. Once a variable has been defined in FTD it can be updated by by Rust code on backend using ftd::Document::set_bool() etc family of functions. Similarly the variable can be updated by window.ftd.set_bool() etc family of functions.

So we have an emerging pattern, we define some variables in FTD with some initial values, and they can be made dynamic by host. If we make this a first class thing, currently we only support changes to basic types like boolean and string, if we support complete access to FTD internals from host, and reliably update the FTD “view” based on FTD data it would be cool.

Library::get() Simplification

Currently we have a technique where we create “virtual” FTD documents, e.g. there is a module fastn for our FASTN static site generator. This module is dynamically constructed with actual values set for all these variables. This is done so we can pass a bunch of data that we have in Rust to FTD files. So we do something like this:

dynamically generating last_modified_on

format!(
    indoc::indoc! {"
        -- record ui-data:
        string last-modified-on:

        -- ui-data ui:
        last-modified-on: {last_modified_on}

     "},
    last_modified_on = fastn::i18n::translation::search(
        &lang,
        &primary_lang,
        "last-modified-on",
        &current_document_last_modified_on
    ),
)

We are setting last_modified_on based on string substitution in Rust using format!() statement.

If we have first class support for setting any variable from Rust, we can keep fastn.ftd file static, with no data in it, and load it to create FTD variables, and then update each of the variables defined by fastn.ftd file.

This means our ftd::Library::get() methods can be simplified. It currently perform two tasks:

  1. Given an ID, it transforms the ID, e.g. it looks up the ID in dependency list, applies complex module resolution logic and reads the final file from disc.
  2. It generates dynamic ftd files like I shown above.

The transformation etc can be done in a sync way. Reading a file itself can be argued to be async, but for many cases it can be considered sync, say if you have an in-memory database of ftd files.

Point is, if we recommended use case 2, it would necessarily mean ftd::Library::get() be strongly async, allow arbitrary complex data lookups, HTTP API calls, database calls and so on. Where as if ::get() is only for loading FTD files, it’s relatively simpler. In many many use cases, say for serving blog or static content, content is small and can be read on application start, with some watcher to keep updating in memory cache on file system changes. In other cases, say FTD working as primary front-end for application, the number of FTD files would be even smaller in general. So it is conceivable to keep entire set of FTD files in memory.

This will still leave out fetching FTD files from say internet. If we want to support that, and there may be some security issues there, but maybe they are solvable somehow and in general we should allow fetching from internet etc.

So while we can not fully make ftd::Library::get() sync, at least we can limit it’s complexity, and not have two ways to set dynamic data.

Reviewing $processor$

So if we have first class support for update FTD data from host languages, then do we still need $processor$?

Our ftd::Library Rust trait exposes another method ::process() whenever FTD interpreter encounters a $processor$ directive. process() computes the data, and passes it to interpreter, and then interpreter continues interpreting the rest of the document.

Technically we can make this also two pass, let the interpreter interpret the entire document in peace (only interrupted by ::get() calls, to be executed every time interpreter comes across an -- import: directive).

We are basically seeing if we can make the whole interpret pass “pure computation”. Now we proved get() can’t be pure yet and we probably will accept that design choice. But what about $processor$? Can we remove it?

One key difference between fastn scenario I described above vs $processor$ is fastn was “well known”. fastn module is well documented (or is going to be), and so Rust code knows what all dynamic variables are there, and Rust can update them.

But in case of $processor$, it can be applied on any user variable defined in arbitrary FTD file. Interpreter can give each of $processor$ variables a default value, go through the document once, and afterwards let Rust update it How would Rust know what all variables to update?

$processor$ to update variable

-- string foo:
$processor$: get-some-data-from-internet

Here we have a processor get-some-data-from-internet, which returns a string, and the string is bound to the name foo in FTD.

Currently when interpreter encounters $processor$ directive, it immediately calls the get-some-data-from-internet (via Library::process()), and initializes foo.

If we want to do it after the fact, we would have to discover what all variables have processor applied.

We have another use case of processor planned:

$processor$ to component

-- ftd.text:
$processor$: get-some-data-from-internet

In this case we are not keeping the output of get-some-data-from-internet in any FTD variable, but instead directly passing the value to the component.

Currently there is no plan to allow host to directly modify the FTD generated DOM, we want to do it via data updates, host only knows about data, updates the data, and UI updates itself.

On the whole it feels to me that creating new API to get variables that have $processor$ or worse allowing host to directly update DOM are not great ideas.

So our interpreter can not really be pure data operation, and it seems the best design is to allow for first class async support and continue on current path.

Object Implementation Details

15th Jan 2022

So Arpita has just implemented object constructors:

-- object obj:
function: console-print
value: Hello World

With this now you can construct JSON objects, eg above will create:

{
    "function": "console-print",
    "value": "Hello World"
}

And then you can pass the object to JavaScript using message-host, eg:

-- ftd.text: click me
$on-click$: message-host $obj

And in JavaScript you can write the consumer:

window["console-print"] = function(v) {
    console.log(v.value);
}

This JS code has to be provided by the “FTD Host”, eg FASTN is one such FTD Host. To create arbitrary functionality, FTD Host interface has to be implemented.

This is the basic. Soon the object constructor will be able to refer to other variables in FTD, eg:

-- object obj:
function: console-print
value: $foo

Here we will use the value, and type of $foo, and that will be inserted in the object.

Objects Are Opaque

One important point to note is that objects, once constructed, can not be introspected from FTD. So you can not do: