Now that we are creating a lot of FPM packages, question of documentation and best practices starting to dictate our features. One thing I have struggled with is how to organize a package.
Till a while back the only ergonomic way was to put everything into one single index.ftd file. We had to do single file else there would be a lot of import statements in each file. Also it interrupts the writing, if you are in the middle of a ftd file and decide to use a new component, you have to temporarily jump to the top of the file, add an import statement and then come back to your original place and use the component.
Now that we have auto import feature, one can easily import more than one module from a package in every file. One would ask people to add a bunch of auto imports when adding a dependency, and we are done.
This means we can now breakup our package into smaller files. So the question is how small a file?
When writing documentation I started with putting the entire documentation in the index file. But then the page is becoming quite long, so I felt I should split the documentation file as well.
I think it would be better if we created a file for each component, and in that file we put the documentation as well as the component source. But when we do the ideal name for the file would be the name of the component itself. Also since ftd only supports importing a module and not individual symbols from the module we would have to end up writing:
-- import: some-lib.com/foo/header -- header.header: something
header.header
etc. What if we allow people to write:-- import: some-lib.com/foo/header -- header: something
module.<whatever is named after module>
. It can be either a variable, record or component. The only main downside is if we see something like above we do not know if header
refers to some imported symbol, in which case I would go review auto-imports
in FPM.ftd
file, or a component defined locally in current file. Since author files usually won’t have components defined or may be one component defined, in general we do not have this issue.
As long as we support any feature like alias, we will have to review FPM.ftd
to see the symbol’s full name. The only way to avoid jumping to FPM.ftd
to understand each component used, is to not use auto import or dependency alias feature, and put explicit import on top of each file.
Let’s hope it is really final, Arpita is now going to implement it.
Every binding is now by reference. So if you define:
-- string foo: hello -- string bar: $foo
foo
and bar
have the same storage in memory and both are names refering to same thing. One can opt out of this by cloning:
-- string foo: hello -- string bar: ftd.clone: $foo
ftd.clone
in due time. Any attempt to modify anything would be governed by the mutable-by
rules. The exact syntax of mutable-by is not fixed but concept is.
We will further allow mutable-by to following package interfect. So e.g.
fpm.ftd
-- boolean dark-mode: false mutable-by: fpm.dev/dark-mode-switcher
fpm.dev/dark-mode-switcher
will be able to modify the fpm.dark-mode
variable. Further anyone that uses such a package will have to still opt in:
-- fpm.dependency: arpita.com/dms implements: fpm.dev/dark-mode-switcher
amitu.com/FPM.ftd
has explicitly granted arpita.com/dms
fpm.dev/dark-mode-switcher
“rights”.So Arpita is working on -- boolean foo: $bar
and we are stuck at the question we discussed on 29th Jan.
-- boolean y: true -- boolean x: $y -- foo: boolean $x: $y boolean z: $y
foo
we are going with the proposal Arpita likes for declaring variable references: boolean $x
meaning $x
is now tied to the rvalue
. If rvalue
changes $x
will change. z
is assigned rvalue
at declaratin and does not change afterwards. Crux of the problem: why do we not always have references? If we do any local change in component will be reflected globally if any variable was to default to a global value. Many times we want defaults set by global, but if inadvertantly every component starts modifying globals it would be a mess.
The specific problem Arpita had in above snippet was now we have different syntax, if global assignments are always reference, but in component its always value then we have an inconsistency which she rightly doesn’t like.
One option is we go with her syntax, <type> $x
means its an alias, <type> x
means it is a assign current value. My syntax was <type> variable x
vs <type> x
. If we go with either of them, does it solve our problem?
The thing is component changing global variable should be exception, but component getting latest value of global should be norm.
Question: why are we then creaitng a new variable in component, and not just using the global variable from inside the component? We want to use global variable as the default value of some component property, but we want callers of component to change that if they want to.
Would readable vs writable semantics help? If we are doing:
-- ftd.row foo: boolean x: $some-global
x
to some-global
, but want callers of foo
to be able to overwrite it. We are not creating a variable to change it in the component after compnent has been constructed. Currently there is no way to declare if a variable is mutable or not. If there was we could have different behaviours: a mutable variable will not reflect changes to global, or maybe a mutable variable will not reflect changes to global after it has been mutated once. But it will never mutate global.
Should we say its always reference and always non mutable. To make it mutable you opt-in, and then behaviour changes. Or we decide if it’s mutable by analysing the body of the component.
Say we use a explicit mutable syntax, and only allow event handlers to modify a mutable variable:
-- ftd.row foo: mutable boolean x: $some-global
-- ftd.row foo: boolean $x: $some-global
How about we go with this bevaviour: we don’t have mutable, any component variable with default value of global will keep reflecting changes in global till it has been modified by the component, once it is modified it starts ignoring global changes. Further, a component can modify global by directly using the global variable in the event handlers.
We will still need reference sytnax: the above the was the default behaviour of normal variables. We will still have <type> $foo
or <type> variable foo
to indicate it takes a variable from caller context and it always modifies that variable.
How about global reference:
-- boolean y: true -- boolean x: $y -- foo: $on-click$: toggle $x
x
or y
as well? We can say we follow the same behaviour for global variables as well, x will continue to track y
till x
was modified. Unless x
was defined as boolean variable x
, in which x
and y
are always the same variables.Identity expression on a variable $x
is simply $x
, any other expression, eg $x * 2
is non-identity (even $x * 1
or $x + 0
are non identity expressions).
If a variable is defined with non identity expression on another variable it is considered a formula and can not be changed. Non identity expressions can not be used as rvalue
of <type> variable x
.
Objection: Variable changes meaning on runtime. For a while it will follow another variable and suddenly it will stop. It would be hard to debug. We can put a warning on console everytime it happens, when a variable changes “nature”, we can even give warning on build time that there are now variables that change their nature.
To understand where Label1’s text is coming from, you know exactly where to look: the formula in the Text property.
This is huge in terms of debuggability of a complex Excel/PowerFX codebase. Everything is pure. There is no concept of time. Only internals of a component can modify a variable, from outside a component you can not.
The problem with them is the same, everything is wrapped in UI components. FTD has a pure data language representation, we are trying to keep data in globals so they can be easily accessed from programs, and components modify these globals, but since any component can modify any global, we have a potential debugging nightmare.
So data only in UI, no globals, and pure expressions over these ui specifi data? Or data first, and UI later to modify the data? In Excel/PowerFX it’s a bit hard to have two “views” that can modify same underlying data. Consider Dark Mode, we model it as a global, fpm.dark-mode
, but in PowerFX it would be managed by a cell, and what if you want to have dark mode switcher in multiple places in the UI? Since they are different components either it would not be possible, or PowerFX will have an escape hatch and will make some sort of singleton variable shared by all instances of the given component. We can do this as well, create “static” or singleton modifier for variables, and share it across all instances of a component.
But then we are still facing all data is in UI problem. We have to identify UI elements using IDs, or name of component if they are singlton. Current clean data modelling will go.
What is we say globals of a module or a package can only be modified by components defined in that module or package? We still allow top level modifications, eg config modifies colors etc, this is not part of component but top level declarations.
If we had way to limit modifications, eg mut(package) boolean foo:
we are saying only current package can modify this foo. Default can be mut(module)
meaning only current module can modify it. And we also allow mut
meaning anyone can modify things. Or we change defaults, mut
means mut(package)
, and mut(pub)
or something to mean everyone can modify.
So we go with the “Final Proposal” with mut
added to limit debugging scope?
-- boolean foo: true mutable-by: package mutable-by: module mutable-by: all mutable-by: some-other-package
mutable-by
can be specified multiple times. package
and module
refer to current package and module respectively, or you can specify any package or specific module. Or you can specify all
to say everyone can mutate this. By default it would be mutable-by: module
.We are going to add boolean expressions in FTD now. Till now we support if
at component level, attribute level, and variable level.
-- foo: if: $something color if $something-else: red -- string bar: yo -- bar: hello if: $something
if
both at component level, and attribute level. In the next example we use if
to control bar
variable. Lets call if:
as section level if
, and foo if $something
as attribute level if
.
Till now we can either do $something
or not $something
, but not higher boolean expressions like if: $a or $b
etc.
We want to allow general purspose boolean expressions now, with and
, or
, and grouping using (
/)
. We will follow Python syntax for that.
What about multiline expressions? We can use multiline expressions for section level if
easily:
-- foo: if: ( $a or $b or ( $something and $something-else ) ) -- foo: if: $a or $b or ( $something and $something-else )
-- foo: color if ( $a or $b or ( $something and $something-else ) ): red
p1
grammar/parser for this. We can always implement the inline versions first, and do the more general syntax later on. Other thought was we created intermediary “formulas” and use them.
-- boolean some-formula: ( $a or $b or ( $something and $something-else ) ) -- foo: color if $some-formula: red
-- ftd.row foo: boolean some-formula: ( $a or $b or ( $something and $something-else ) ) \--- ftd.text: color if $some-formula: red
-- boolean some-formula: $a or $b or ( $something and $something-else )
-- ftd.row foo: \--- boolean some-formula: $a or $b or ( $something and $something-else )
---
. With component level variable we never need multi line header key or value to support multiline boolean expressions. Which means we won’t have to modify our p1 parser and grammar remains simpler. Arpita thinks even if we use multiline value or caption, we must use our continuation syntax:
-- ftd.row foo: boolean some-formula: > $a or $b or ( > $something and $something-else > )
-- boolean foo: true -- t: val: $foo -- ftd.text t: hello boolean val: color if $val: red
$foo
, should $val
change? This is currently implemented as pass by value. We are planning a syntax in future:-- t: val: $foo -- ftd.text t: hello boolean variable val: color if $val: red
val
as boolean variable
instead of boolean
as in last example. With x variable
, the we will implement pass by reference semantics. We call this feature higher order variable. Arpita proposes this syntax:
-- t: val: $foo -- ftd.text t: hello boolean $val: color if $val: red
One open question for higher order variable is should this be allowed:
-- t: val: true -- ftd.text t: hello boolean variable val: boolean $val: color if $val: red
true
but $val
need to bind to some existing variable, and we only have a value. We can say we would allow it, and any modifications to $val
would be lost, semantically its modifying that true
, but since nothing else is bound to it the modification is not visible outside. With this we do not have any major objections to higher order variable.
So we have two possible variables, changes in parent reflect in child and child in parent when we use higher order variable. Or change in neither side is reflected in other side, when using normal variable (pass by value). The default of pass by value seems bad, as if a parent variable is modified it’s value is updated everywhere it is directly referenced but not in children:
-- ftd.row foo: boolean o: true \--- bar: o: $o color if $o: red $on-click$: toggle $o
$o
is changing, but o
is not. This feels wrong. Shouldn’t people just write:
-- ftd.row foo: boolean o: true \--- bar: o if $o: true o if not $o: false
I am not saying pass by reference should be default, else any changes done by bar to $o
will always be visible to parent. I mean we can do that. But I feel there should be a third choice where changes go from parent to child always. If the child component is modifying the variable that modifications is not reflected outside, but if not then it is.
If the variable was never modified by a component then the value passed from outside could always be reflected and there is no concern. But if component is updating a variable then what to do? We can say if variable is never modified by component then we do pass by reference, else we do pass by reference or value depending on if component has defined variable as higher order or not.
Arpita says if we do this it would be hard to explain as the behaviour of a component will suddenly change if they add an event handler that modifies a variable.
What to do?
MOUSE-IN
Behavior Change$MOUSE-IN
which can be used anywhere inside a component definition without declaring, and it is true if the mouse is hovering over the current component. This is limiting, what if you wanted to use this variable from outside a component? Or what if you want to be more precise, within a component if you want to do something if the mouse is within a specific child in the component?
If you want to emulate the old behaviour you will have to do this:
-- ftd.row foo: boolean mouse-in: false \--- ftd.text: yo $on-mouse$: $mouse-in=$MOUSE-IN
$on-mouse$
that will be triggered when mouse enters or leaves a component. Then we can do the following as well:-- ftd.row: boolean mouse-in: false border-width if $mouse-in: 1 \--- ftd.text: yo $on-mouse$: $mouse-in=$MOUSE-IN
$MOUSE-IN
.We can already create comics using FTD. But we need is more succinct syntax to represent a bunch of position, scale, rotation related attributes into a single short-hand notation, and eventually give a drag and drop interface that updates the position etc shorthand notation.
-- panel: d: 0, 0, 1 \--- girl-1: pose: flying d: 0, 100, 1.5 a: left-right \--- girl-2: pose: flying a: bottom-top, 0-1, left-right, 1-2 \--- girl-2: Hello Everyone! We are Power puff girls pose: flying a: 2- \--- girl-3: pose: flying a: right-left, 1s
d:
could be dimensions, position, orientation, scale etc. a:
could be animation. We can even have double representation, eg a separate place where all the dimensions and animations are put for the entire scene, so rest of the code only shows the character names, poses, dialogs etc. If we do this, we can make the “source” of comic almost as readable as the rendered comic itself. And this technique is not limited to only comics, any thing that requires spatial arrangement, but is can otherwise be described better in text, say an diagram, model of a house and so on can all benefit from such a representation.
We tend to think that such things can not be represented in text, but maybe it’s only because the optimal representation has not been created yet.
The advantage of text is tremendous. We can use Git etc, the only technology worthy to be called collaborative, Google Docs, whose entire reason for existence is “word processor collaboration”, can not scale to dozen collaborators, forget 100s or tens of thousands, which git can easily. None of the Figma’s, Adobe’s offerings, random comic generation tools, equation editor, animation creator, etc work with reasonable number of collaborators, who would create an account with every diagramming, comic generation, image editing tool you use. There is just too much fragmentation, entire companies on individual tools, but your workflow needs all of them. Git can solve that if there was a text file representation for each need.
Was discussing function definition stuff with Arpita and I am quite happy with the design in general. We have can pass variables to functions by reference (so they can be modified by the function), or value. Function decides if the passed argument is by reference or value. Beyond these referenced variables, a function can not modify any variable. Function can read other global variables but can’t modify them.
I also like the either a function call is used as a. event handler, in which case function must have a return type of void
or null
, or maybe we can even call them event-handler to make the distinction clearer, or b. formula, a function whose output is “bound” to a name, every time any of the arguments of the formula, or any global reference in the formula is changed, the value of the bound name is also recomputed and changed.
Error handling with result and option being special data types, and our if
expression handling them is also still making sense after a few days of deliberations.
What about async
? What if a formula needs to depend on say local store, or some API? Should we support it? I do not yet have a good answer to this. I feel like saying just switch to “host language” for such things, where you get full first class language, instead of FTD. But do I want to say FTD is not a first class language? Where to draw the line? We can obviously not give all the host capabilities to ftd, I mean we discussed reading from API, but what about reading from USB or file system or Windows Registry, etc etc? We are going to have to depend on host functions to give access to such features.
We can not be a single language. We are UI language. Data manipulation and arbitrary computation (within limits) on the data should be supported first class, but we can not go for complete replacement. We also have to live side by side with another “host” language.
But is host language “user programmable”? Or is it only for giving users building blocks, that they can then connect together? This drawing a line between what must forever remain host language domain, and what must be handled by ftd is tricky to resolve for me yet.
Now that we have basic dynamic components working, we need to think of syntax for more actions. We currently has “SQL-esque” commands, eg increment $foo by 1
etc, instead of function like syntax: $foo.increment(by: 1)
. We have to figure out what we want our eventual syntax to look like.
Some command operations:
increment $foo by $incr clamp 10
.$foo=<new value>
.toggle $foo
insert $el into $list at end
or at front
or at <position>
.clear $list
or clear $optional
remove from $list at $pos
-- ftd.text: hello $on-click$: increment: what=$count, by=2 > what: $count > by: 2 > clamp: 10 -- ftd.text: hello $on-click$: increment: what=$count, by=2, clamp=10
ftd
will be ftd.<foo>
. Arguments will always be named, no positional arguments. We will use =
for arguments that come on same line, and :
for arguments that come in subsequent lines. Functions can define caption
type, in such cases, other arguments can not come in caption line.-- void increment: integer variable what: integer by: 1 optional integer clamp: integer new = $what + $by if $clamp && $new > $clamp { $new = 0 } $what = $new
void
for functions that are only for side effects, e.g. to be directly called from $on-*$
. For other types, eg integer it will appear to look like an integer variable definition, but since it takes arguments, it’s a function. The last expression can be considered the return type of a function, but we will also support return <value>
. Arguments are specified in usual manner.
integer variable what
, which is to indicate that here we are not accepting a value, but the reference to some variable, which would be modified by this function.Some functions can be defined as pure, they can not take higher order variable, and can only be called from variables defined at global levels. Formulas can depend on other variables, and when the underlying variable changes, the value of the formula changes automatically.
-- integer length: 20 -- integer foo: area: length=$length, width=20
foo
is considered a formula. In such cases further updates to foo
is not allowed. If $length
changes, foo
would be auto-updated.Variables change from events. What kind of errors: division by zero. Parsing error, e.g. if someone is trying to enter an integer in an input field, and they type some non digit. Or say if we are reading some data from JSON, and we are expecting an integer but we get a string, or the key is missing.
Among these errors I listed, divide by zero is special, in being an operation that we usually don’t want to make “error check before proceeding”, for others we can ensure all functions that can fail return some value that forces one to handle failure possibility. Forcing error check for divide by zero will force all mathy code to become too verbose.
Any error that happened during initial page load is “fine”, we simply bail, show error message and let author figure it out. This is equivalent to “compile time”, and we want people to not store invalid documents. But runtime errors we can not avoid.
Result
And Option
Result
and Option
types are there. It kind of means we need some level of generic support. We can have special handling for Result and Option types without generic. So we can write something like:
-- integer result to-integer: string value:
to-return
that returns integer result
, (for list it would be integer list result foo:
).if <var-name> = <result expression> { success clause, can use <var-name> } else <error-name> { error clause }
-- void foo: let i = if num = to-string: value=hello { num } else error { console.log(error) }
-- optional integer foo:
optional values as well. In such cases, the else
block won’t take error
variable name, it will always be null. Error type will only be string. In future when we have enums, we can also do:
-- string enum Errors: invalid-integer: Value Is Not A Valid Integer empty-value: The value was empty string overflow: The value is too large to fit into integer type. -- void foo: let i = if num = to-string: value=hello { num } else Errors.invalid-integer { console.log(Errors.invalid-integer.message) } else Errors.empty-value { console.log(Errors.empty-value.message) } else error { console.log(error) }
enum
.<type> result
is not an or-type
or-type
is our name for enum
in Rust. In future we will have match
statement for handling or-type
, but we are not going to treat result
and optional
as or-type
, so match
can never work with them.
return 1
to indicate success, and fail "whatever the message
to return failure message. We will also support let num = to-string? value=hello
, here ?
acts like Rust.default record
-- void do-get: fetch-people(after-get, on-error) -- void fetch-people: success(person list) -> void: failure error->void: ftd.http-get("https://foo.com", success, failure) -- void after-get: person list people: -- void on-error: error error:
Till now when a page used to get rendered from backend, it used to contain the entire component hierarchy, and front-end event handling etc only changed some of the DOM attributes, and visibility. If there was an element whose visibility was affected by an if
clause, it used to be always included, and set as hidden and when needed we used to make it visible. We never created DOM node till now as part of event handling. This means it was not possible for us to have say a list, and add elements to the list from any of our event handling code.
Yesterday Arpita implemented support for dynamically constructing DOM when say we add elements to a list.
–Consider this:
-- ftd.column: spacing: 10 -- ftd.row: width: fill spacing: 10 -- ftd.input: placeholder: Type Something Here... width: 250 border-width: 2 padding.px: 8 $on-input$: $query=$VALUE -- ftd.text: Add if: $query is not null color: $inherited.colors.text background.solid: $inherited.colors.background.base move-down: 4 padding.px: 5 $on-click$: insert into $strings value $query at end $on-click$: clear $query -- ftd.text: Clear color: $inherited.colors.text background.solid: $inherited.colors.background.base move-down: 4 padding.px: 5 $on-click$: clear $strings -- ftd.text: if: { $query != NULL } color if { $ftd.dark-mode }: $inherited.colors.text You have typed: {value} -- ftd.text value: $query color: $inherited.colors.success.text color if $ftd.dark-mode: $inherited.colors.text background-color if $ftd.dark-mode: $inherited.colors.background.base padding if $query is not null: 5 role: $fpm.type.copy-large -- show-data: $obj $loop$: $strings as $obj
Arpita got this brilliant insight that we can create one instance of each such component and keep them hidden, and when we have to create any new instance of any of these components we clone that DOM node, and set it up with correct data.
It turned out to be as simple it sounds, and entire insight to merge was 3-4hrs! I was worried about JS size increase etc, and none of that happened. So little code, such a massive impact.
Now that we have support for creating nodes dynamically, we are almost complete UI solution, and FTD should be considered an alternative anywhere you would consider using say React for creating UI.
Some variables, like fpm.document-title
, could be set by more than one ftd documents, each document is telling us what is the document-title for that document by setting fpm.document-title
.
But we allow authors to import any document from any other document, barring cycles, so these variables can get clobbered. Say if there is a.ftd
and b.ftd
and former wants document title to A
and later to B
, they can do both: -- fpm.document-title: A
and so on. But what if a.ftd
looks like this:
-- import: fpm -- fpm.document-title: A -- import: b
b.ftd
would be imported after a.ftd
has updated fpm.document-title
, and it will overwrite fpm.document-title
. We can argue this is fine at some level and user is aware of things. Further there is another concern, what if fpm.document-title
was optional, and the idea was if this is not set, we pick the title from first heading of the document. And say this worked out fine, so a.ftd
will not want to fpm.document-title
, but say it does not work for b.ftd
so it wants to explicitly set it. But then a.ftd
imports b.ftd
. We can not solve by import order etc.
Proposal:
fpm.ftd
file-- optional string document-title: $main$: true
$main$
, then only the changes done by the “main document” would be kept. FTD interpreter knows whats the main document that is being interpreted, and it knows when it is interpreting one of its dependencies (via import). We can further allow $main$
during variable change also to allow non main documents to update variable still, eg a config.ftd
or some file which is only for setting project variables.
config.ftd
-- import: fpm -- fpm.site-title: AmitU's Blog $main$: true
a.ftd
-- import: ftd-dev.vercel.app/config $main$: true -- import: b
config
as “main”, but not b
. This would be applicable for both variable overwrite, and for adding a an element to a list, either can use “$main$”.
$main$
variable can only be modified by $main$
, 3. and 4. decide when a document is considered $main$
$main$
while updating the variable$main$
document$main$
document is importing some document it can declare those document as $main$
as wellb.ftd
:-- import: fpm -- fpm.document-title: B -- ftd.text show-title: $fpm.title
a.ftd
was:-- import: fpm -- fpm.document-title: A -- import: b -- b.show-title:
show-title
will show A
and not B
even though if you look at source of show-title
in b.ftd
it appears it is modifying a variable and using it right away. This is not particularly confusing because a variable could have been updated multiple times, even within the same file:
-- import: fpm -- fpm.document-title: B -- ftd.text show-title: $fpm.title -- fpm.document-title: New B
fpm.document-title
is not used by show-title
, but the “eventual value”.$ref
syntax-- import: bar -- object foo: key: $bar.baz
bar.baz
is 20
(integer
), it will generate this:{ "key": 20 }
message-host
to only accepts objectsmessage-host
. It currently accepts either a string, which is the name of function exposed by the FTD host, or it can be an object, that has to define a key function
, which will be the function. In general I am not fond of “string values”, because if you make a typo, ftd interpreter can’t help you. The host functions are provided by host language, and ftd compiler has no idea about host functions. Ideally FTD compiler should have some knowledge about capabilities provided by host, maybe some sort of schema and type definition?
In the meanwhile, accepting both string and object, I now consider an anti-pattern, we should only support well known object constructors:
-- import: fpm -- ftd.text: Switch To Dark Mode $on-click$: message-host $fpm.switch-to-dark-mode
message-host switch-to-dark-mode
, where switch-to-dark-mode
was a string, but $fpm.switch-to-dark-mode
is a reference to something FTD interpreter can verify.$processor$
, async
, Library::get()
and Document::set_*()
One of the things that has emerged since yesterdays brain storm with Arpita is we need first class support for updating FTD variables from “host languages”.
You can say we have already two host languages, Rust and JavaScript. Once a variable has been defined in FTD it can be updated by by Rust code on backend using ftd::Document::set_bool()
etc family of functions. Similarly the variable can be updated by window.ftd.set_bool()
etc family of functions.
So we have an emerging pattern, we define some variables in FTD with some initial values, and they can be made dynamic by host. If we make this a first class thing, currently we only support changes to basic types like boolean and string, if we support complete access to FTD internals from host, and reliably update the FTD “view” based on FTD data it would be cool.
Library::get()
Simplificationfpm
for our FPM static site generator. This module is dynamically constructed with actual values set for all these variables. This is done so we can pass a bunch of data that we have in Rust to FTD files. So we do something like this:last_modified_on
format!( indoc::indoc! {" -- record ui-data: string last-modified-on: -- ui-data ui: last-modified-on: {last_modified_on} "}, last_modified_on = fpm::i18n::translation::search( &lang, &primary_lang, "last-modified-on", ¤t_document_last_modified_on ), )
last_modified_on
based on string substitution in Rust using format!()
statement. If we have first class support for setting any variable from Rust, we can keep fpm.ftd
file static, with no data in it, and load it to create FTD variables, and then update each of the variables defined by fpm.ftd
file.
This means our ftd::Library::get()
methods can be simplified. It currently perform two tasks:
The transformation etc can be done in a sync
way. Reading a file itself can be argued to be async
, but for many cases it can be considered sync, say if you have an in-memory database of ftd files.
Point is, if we recommended use case 2, it would necessarily mean ftd::Library::get()
be strongly async
, allow arbitrary complex data lookups, HTTP API calls, database calls and so on. Where as if ::get()
is only for loading FTD files, it’s relatively simpler. In many many use cases, say for serving blog or static content, content is small and can be read on application start, with some watcher to keep updating in memory cache on file system changes. In other cases, say FTD working as primary front-end for application, the number of FTD files would be even smaller in general. So it is conceivable to keep entire set of FTD files in memory.
This will still leave out fetching FTD files from say internet. If we want to support that, and there may be some security issues there, but maybe they are solvable somehow and in general we should allow fetching from internet etc.
So while we can not fully make ftd::Library::get()
sync
, at least we can limit it’s complexity, and not have two ways to set dynamic data.
$processor$
$processor$
? Our ftd::Library
Rust trait
exposes another method ::process()
whenever FTD interpreter encounters a $processor$
directive. process()
computes the data, and passes it to interpreter, and then interpreter continues interpreting the rest of the document.
Technically we can make this also two pass, let the interpreter interpret the entire document in peace (only interrupted by ::get()
calls, to be executed every time interpreter comes across an -- import:
directive).
We are basically seeing if we can make the whole interpret pass “pure computation”. Now we proved get()
can’t be pure yet and we probably will accept that design choice. But what about $processor$
? Can we remove it?
One key difference between fpm
scenario I described above vs $processor$
is fpm
was “well known”. fpm
module is well documented (or is going to be), and so Rust code knows what all dynamic variables are there, and Rust can update them.
But in case of $processor$
, it can be applied on any user variable defined in arbitrary FTD file. Interpreter can give each of $processor$
variables a default value, go through the document once, and afterwards let Rust update it How would Rust know what all variables to update?
$processor$
to update variable-- string foo: $processor$: get-some-data-from-internet
get-some-data-from-internet
, which returns a string
, and the string is bound to the name foo
in FTD. Currently when interpreter encounters $processor$
directive, it immediately calls the get-some-data-from-internet
(via Library::process()
), and initializes foo
.
If we want to do it after the fact, we would have to discover what all variables have processor applied.
We have another use case of processor planned:
$processor$
to component-- ftd.text: $processor$: get-some-data-from-internet
get-some-data-from-internet
in any FTD variable, but instead directly passing the value to the component. Currently there is no plan to allow host to directly modify the FTD generated DOM, we want to do it via data updates, host only knows about data, updates the data, and UI updates itself.
On the whole it feels to me that creating new API to get variables that have $processor$
or worse allowing host to directly update DOM are not great ideas.
So our interpreter can not really be pure data operation, and it seems the best design is to allow for first class async
support and continue on current path.
So Arpita has just implemented object constructors:
-- object obj: function: console-print value: Hello World
{ "function": "console-print", "value": "Hello World" }
message-host
, eg:-- ftd.text: click me $on-click$: message-host $obj
window["console-print"] = function(v) { console.log(v.value); }
This is the basic. Soon the object constructor will be able to refer to other variables in FTD, eg:
-- object obj: function: console-print value: $foo
$foo
, and that will be inserted in the object.-- integer f: $obj.value
$object.value
is an integer
, this may seem “reasonable”, but we do not allow it. At least for now. We are doing it for performance reason one can say, also we are lazy, we do not yet have time to think though the implications of it. We are building this feature to pass data to JavaScript for now, so we are committing to only the part of design that is minimally needed.
{ "object": { "function": "console-print", "value": "$foo" } }
$foo
pattern, and resolve them. We can also have \$foo
to escape this behaviour. Other option is to keep track of references:
{ "references": { "value": "$foo", "foo.bar[0].baz": "$baz" }, "object": { "function": "console-print", "value": "$foo" } }
references
we have “JSON-paths” to every key that is a reference. Its easy to update the object by iterating it. We will see which ends up being easier to implement.
This is why FTD is trying to be a common language that will work everywhere, it will be responsible for UI and all the core part of functionality, like search, API access etc, has to be provided by host.
-- object console-print: string message: function: console-print message: $message -- ftd.text: click me $on-click$: message-host console-print: > message: clicked! -- ftd.input: $on-input$: message-host console-print: > message: $VALUE
-- object get-user-info: string username: $process$: http url: https://github.com/api/user-info method: POST api-key: $api-key -- user-info info: $processor$: get-user-info username: amitu
message-host
can now pass parameters to FTD Hostmessage-host
till now used to accept a function name, and ftd.js
used to call that function.-- ftd.text: click me $on-click$: message-host foo
window.foo()
. But there was no way till now to pass arguments to that function. With the latest changes done by Arpita now we can pass arguments as well:
-- object arg: function: what-ever-function name: jack field: value -- ftd.text: click me $on-click$: message-host $arg
window.what-ever-function
and pass it following object:{ "function": "what-ever-function", "name": "jack", "field": "value" }
name: $name
. We will also make function
optional, if it’s not passed we will assume the name of object, in the above example it was arg
.$VALUE
on ftd.input
, $on-input$
and $on-change
$VALUE
special variable so now you can do:-- optional string query: -- ftd.input: placeholder: Type Something Here... $on-input$: $query=$VALUE -- ftd.text: $query
$query
. See a demo here. We have two events, $on-input$
, which happens as soon as a input is edited, on key press, and $on-change$
, which happens when the user presses Enter after inputting, or if the input field loses focus.
The $VALUE
special variable is only visible to the event handler of the above two events.
value
-- ftd.input: value: $query $on-input$: $query=$VALUE
value
bound to a variable, every time the variable changes the value of the input field would be updated. So you can control the value of one input field with another for example.default
-- ftd.input: default: $query
default
will be used to bind the value of an input field when the input field is initialized, and any further changes done to $query
would have no effect on the input field.type
type
parameter so input field can be used to get username
, which hints the browser to use Password Manager etc, email
, password
etc, similar to how HTML input works.Many sites show a table of content somewhere, usually on the right, or sometime at the beginning of the article after the intro etc. This requires us to query the current page for data. Heading hierarchy is one such data. List of images, tables, diagrams etc could be other such query. Maybe even list of all foot-notes, links etc could be useful as well.
We have “region” to identify some of these things. The query should allow us to query by region, and authors should use regions properly to set this up.
One way to query things is package-query feature that we are working on fpm
. But that is largely for querying information from the entire package. And while one can use the same mechanism to also query for information in current page, it puts a dependency on fpm
. Since ftd has access to information in current page, such a query can be conceivably be easily provided by FTD as well.
What would be the result to these queries? Since these queries are to power intra page linking, they should return #hash
for that specific DOM node. We auto generate id
for each heading as of now. We have to start generating IDs for every interesting region.
So the query could return list of hash, title pairs. Each region will have to be associated with some sort of text as well. In case of image, table etc it could be caption. Else it the name of the object can be the title, e.g. for image we will just show “image”, or even “image 1” etc.
What about heading tree?
We already do heading tree detection, we can return a nested data structure based on our query.
What all to include? We can pass a list of regions that we are interested in.
It has to be a $processor$
. We can call it page-query
, package-query
for entire package, page query for just the page.
-- ftd.toc-item list headers: $processor$: page-query regions: h1, h2, h3, h4, image, table
ftd.toc-item
itself is a record:-- record toc-item: string title: string url: toc-item list children:
Consider this journal page itself, there are a lot of future TODO like headings, many announcements of some feature RELEASE and so on. Now imagine if there was some way to have a selector where you click on TODO button and only the posts with TODO tags are visible.
How would we do that? One easy way to do it is to create components like this:
-- optional string current-tag: -- ftd.text h1: $title caption title: string tags: if: $current-tag is null | $tags contains $current-tag
|
, the or
expression support.toc-item
does not have tags
. region
is a common property, available on all ftd elements. Maybe we can add tags
on all of them as well, and include tags
in toc-item
? We should also add region
to toc-item
.Address ::= SEQUENCE { street-address UTF8String, country UTF8String -- see a note below, postal-code UTF8String } ((WITH COMPONENTS { ..., country ("USA"), postal-code (PATTERN "[0-9]#5(-[0-9]#4)?") } | WITH COMPONENTS { ..., country ("Canada"), postal-code (PATTERN "[0-9][A-Z][0-9] [A-Z][0-9][A-Z]") } ))
record
, and it has three fields, and for postal-code
they have two different constraints applied, depending on the value of country
field, e.g. if country
is USA
, then postal-code
must match the given pattern and so on. This is quite interesting. We have minimal form handling support now, so user is going to type stuff, and it would be good if we can chose to not accept invalid values.
Similarly for components and theme configuration variables theme and component authors should be able to place some restrictions on the values passed to them.
How could could it look like? First of all it has to be done at both top level, and inside component level. So like -- container:
, which can equally well work with --- container:
, meaning it should not have nesting. If we look at ASN1 example there is clear nesting. I mean if there was no nesting it would be easier to design. Lets see.
Say:
-- string foo: hello -- check foo for person: \--- title: regex: <some regex>
check
keyword. We could have called it constraint foo
as well, but that’s longer to type and I keep forgetting the spelling. We can do it on record as well:
-- record foo: string title: \--- check title: regex: <whatever>
-- ftd.text foo: caption title: \--- check title: regex: <whatever> \--- check p: check: foo
regex
. But we can have max-length
, min-length
, starts-with
, ends-with
, is-lower-case
, is-title-case
, contains
/at-most-one
and so on. Maybe even unicode-range
stuff we use for font-face
. Similarly integer
can be min
, max
, is-odd: true
, is-prime
? etc.
contains
, min
/max
etc can also refer to other variables. How would we disambiguate global vs local variables by same name? We won’t. Don’t overwrite shadow variables, it’s not a good idea, we will put a lint for that.
How would we know if the value is failing? In some cases we can create errors, so if an invalid value was part of a document we can refuse to save the document etc. But sometimes, e.g. when I am say changing a boolean
, and based on new value of the boolean
a bunch of variables that were initially valid are no longer valid. How do we deal with that?
One option is we accept the error case, and for each variable we have a special attribute .error
, which can be optional string
, will be null
if the variable checks pass, else it would be the user visible error message.
Error message in what language? We can’t pretend there is only one language now can we? Especially when first class translation support is our one of shining features in fpm
? Problem for tomorrow me!
-- check
can specify some error message as well if the check fails. What if there are more than one errors? Make it .errors
, like Django forms? Auto message, e.g. failure of min
, we can create an error message in right language.
markup
support, we can create string templates in every language for some string, and use variable substitution etc.-- ftd.text welcome-message: string who: text if $fpm.lang == en: Hello, {$who} \--- text: if $fpm.lang == hi: नमस्ते {$who}, आपसे मिल कर बहुत ख़ुशी हुई.
$on-click$: message-host <>
Arpita just implemented message-host
event handler support. Now the “ftd host” (we will rename ftd::Library
to ftd::Host
terminology soon), can define a JavaScript function (and in future when we support iOS etc, register Swift function etc) and call it as event handler from FTD documents:
foo()
host methodwindow.foo = function() { console.log("foo"); }
foo()
host method-- ftd.text: call foo $on-click$: message-host foo
Was writing a CR about “fpm actions”, and my initial thought was to expose form errors using special variables, but then I realised we do not have a mechanism to register for changes in special variables.
Variable change can only happen on client (it can still be triggered by server and sent to client, eg on WebSocket). We are going to have fpm.js
file to handle those variables. We already have methods in ftd.js
to update any variable, and already have variable dependency tracker that updates the rendered DOM tree in an efficient way.
One simple answer is to not have a single virtual document, but have a bunch of document, maybe even one document per variable in extreme case. So we only import the document that we want to use.
We will also have a linter that will complain if a document is imported but not used, (and therefore we have import: foo as _
syntax).
.get()
method, and FPM is putting a lot of interesting logic in it, so this part of design is proving fruitful. Then we need special documents for “generated” variables, and ftd host will take care of updating them on client if needed. FPM also exposes processors, and that’s also working out quite well, there are very FPM specific processors.
This does mean there is issue with portability of FTD documents between different “FTD Hosts”, would we have more than one hosts of FPM would be the only host?
We have at least one contender for next FTD host: arbit pages. If you want to use ftd to power say your homepage (without using fpm-repo
), and you must consider, you are going to need your own variables with your own way of fetching documents and your own
Arpita has implemented a unified syntax for component creation.
-- ftd.row foo: spacing: 20
-- ftd.row foo: spacing: 20
-- integer bar: 20
<type> <name>:
, both for variables, records and components, as well as for fields and argument definition in records and components.-- ftd.column foo: ftd.ui a: integer size: 10 \--- a:
ftd.ui
, the argument a
for component foo
here has type ftd.ui
. Any ftd kernel component can be passed to someone expecting a ftd.ui
.
-- foo: a: ftd.text: hello
ftd.text: hello
to foo
as a
. We have also implemented what we call “continuation” to pass parameters:-- foo: a: ftd.text: hello > size: 20 size: 12
size
to both foo
, size: 12
goes to foo
, and to ftd.text
, > size: 20
means we are continuing from the previous line, and hence size: 20
is passed to ftd.text
. We can do arbitrary nesting this way.
-- foo: a: foo: > a: foo: >> a: ftd.text: hello >>> size: 12 >> size: 30 > size: 20 size: 12
$var
Everything Is DoneWe finished implementation of $ for all variable.
-- ftd.text foo: text: some text
-- ftd.text foo: text: some text
-- integer bar: 20
-- ftd.row foo: ftd.text child: -- foo: child: ftd.text: hello
child
of type ftd.text
.ftd.row
, but the component derived from a ftd.row
is no longer row-like
? Why do I want a row specifically? It kind of does not make sense? Maybe I want a container, in which case I may be adding children to that container, which means I have to have generic types like ftd.element
, which accepts any ui element, or ftd.container
, which accepts a container. What about ftd.image
? What if I want to say give me an image. I can pass any instance ftd.image or instance of a component created directly or indirectly from ftd.image. But what about a row that wraps an image? Should that row not be accepted by a component that is expecting image? Why would any component want to accept an image specifically?
fpm
.try.ftd.dev
play.fifthtry.com
to try.ftd.dev
,now that ftd
docs are moving to ftd.dev
. They will still use FifthTry, just the domain would be custom.$var
Everything UpdateArpita has partially implemented $var
everywhere change. As of now, global variables can be referred using $foo
syntax, no need to use ref
. Also @foo
syntax is gone, any $foo
is editable as well.
What she is working now is external variables, you can define a variable on top of a component:
-- ftd.row: $foo: boolean with default false
boolean with default false
syntax and use boolean $foo: false
or even $foo: false
to declare a variable.Wrote about my thoughts on cleaning up FTD public crate api.
detached
support proposaldetached
components: components that are not part of UI when created, and added a new include
keyword to include any detached component in “current container”.We have been keeping a personal journal in FifthTry internal documentation, but it would be better if it moved all FTD related things here for everyone’s benefit.
We kind of believe in writing down our thoughts during all meetings as much as we can, etc, a lot of stuff we write in change requests, linked from the Roadmap page. But since they are all scattered across different CRs, its not easy to keep up with what is going on with the project, so we kind of want to duplicate or link here what is going on.
We also want to create a loom recording after the fact, just to capture a few more things we might have missed when writing it down.
-- var show: true -- foo: \--- ftd.text: Hello 👋 if: show \--- ftd.text: World 🌎 -- ftd.text: Click here! $on-click$: toggle show -- component foo: component: ftd.column open: true append-at: some-id \--- ftd.text: Title \--- ftd.row: spacing: 20 id: some-id
-- ftd.scene: height: 400 -- box: left: 20 top: 20 width: 500 height: 300 -- text: `ftd.main` (`ftd.column`) left: 290 top: 323 -- box: left: 40 top: 40 width: 460 height: 200 -- text-box: Title left: 55 top: 55 -- text: `ftd.row: some-id` left: 300 top: 88 -- box: width: 430 height: 80 top: 110 left: 55 -- text: `foo` top: 215 left: 457 -- text-box: Hello 👋 top: 130 left: 70 -- text-box: World 🌎 top: 130 left: 200 -- ftd.text: Click Here! top: 250 left: 45
--- ftd.text: Hello 👋
and --- ftd.text: World 🌎
are two external children of component foo
and foo
is open at --- ftd.row:
. Earlier we used to wrap all the external children in a column component (Why not row? no good reason just that column is used a lot) and then attach this wrapper component to the desired place. So this created a problem here. Instead of children appearing in a row-wise fashion, it appears column-wise.
Also, open container ftd.row
has spacing: 20
which means all its children would contain spacing from each other, except the first visible one, but here it has only one wrapper child, so no spacing.
data-ext-id
) that has the same value in all the external children and during reparenting, access children using this attribute.As we know there should be no space for the first element, so now that the first child has vanished, the second child should act as the first one and space should be removed. (And obviously, when the first child becomes visible again, the second one should restore the spacing.)
Here’s the solution. We have added a new attribute (data-spacing
) in the parent node that contains the spacing value. Every time this kind of event occurs, we reevaluate spacing for its children.
Consider the following code:
-- ftd.image: src: https://www.w3schools.com/cssref/img_tree.gif link: https://www.amitu.com/fifthtry/amitu/
<a>
tag and for image we use <img>
tag. <img>
tag was overriding <a>
. Basically we were attaching href
on top of img
tag.<img>
tag and make it child of <a>
.