Florian Reuter's Weblog

2011-08-31T18:09:00.009+02:00

ACID test for Absolute Positioned Frames.

Based on the ACID test for Absolute Positioned Tables I created another test for Absolute Positioned Frames (APFs). Make sure you have the Ahem font installed. The test document renders as follows:

Again, here is what e.g. LibreOffice.org makes out of it:

Not to mention again that APFs are important for business documents since they are used in letter heads etc.

2011-08-29T23:38:00.010+02:00

An ACID test for Absolute Positioned Tables

Inspired by the first ACID test ACID1 also called Box Acid Test I created a simple document to test the handling of Absolute Positioned Tables (APTs). APTs are used a lot in letter heads of e.g. business documents.
The test is very simple. It uses the Ahem font from the CSS test suite. Make sure you have the font installed on your machine in order to run the test.
When successful this document is rendered as follows:

Here is what e.g. LibreOffice (3.4.2) makes out of it:

I really like these kind of tests because they are very visual and it is quite easy to understand whether the test worked or not.

2011-08-11T10:42:00.009+02:00

Advances in the layout engine

I started implementing "Shape" support in my layout engine. What's so special about Shapes is the fact that the text needs to flow around them when wrapping is enabled. Shapes are used a lot e.g. to construct letterheads etc. Good Shape support is absolutely crucial for business documents.
Here is my sample document I used for testing wrap03.docx:

It shows Shape objects with "tight" wrapping and the different wrapping modes as defined in 20.4.3.7 ST_WrapText (Text Wrapping Location) of the OOXML specification:

both (Both Sides) Specifies that text shall wrap around both sides of the object.

left (Left Side Only) Specifies that text shall only wrap around the left side of the object.

right (Right Side Only) Specifies that text shall only wrap around the right side of the object.

I'm really happy to have this key feature in the layout engine. Just to show how difficult this is to implement take a look at what Google Docs makes out of it: wrap03.docx.

2011-02-07T16:30:00.004+01:00

Just came back from FOSDEM. Felt really good to meet the “usual suspects” again. Thanks for the great weekend!

I also had a chance to talk with Jos about ODF Web and ODF Collaboration. Jos gave a great talk about his ODF Web Javascript Framework which emerged from his ODFKit efforts.
Jos had a very important slide in his talk which echoed my own believe: NO CONVERSION! This principle guided the design of his ODF Web Framework. NO CONVERSION simply means that Jos does not try to heuristically (aka lossy) map ODF to HTML and then map HTML heuristically (aka lossy) back to ODF. Instead Jos decided to have a clean 2-tier architecture which cleanly separates the content- and the view layer: ODF is content and HTML is the view. I think that’s the right approach. Even more: I think if you start adding “smart conversions”/”heuristics” and other “intelligent mappings” things will get ugly sooner or later. [And from my experience on OpenOffice.org filter hacking things will get messy sooner than you like. Always keep Murphy’s law in mind: What can go wrong will go wrong!].
We also had a chance to talk about Operational Transformation (OT) in the context of ODF. I tried to argue that what is really missing in ODF is a list of “atomic changes” a user can make to an ODF document. If we had this list of “atomic changes” we could build a transformation on top of it. For OT it is very important that you have “atomic” operations, since you need operation transformations for every pair of operation. E.g. if you have |OPS| operations you need |OPS x OPS| transformations. So keeping |OPS| small is quite important!
Assembling the list of atomic operations is a lot of work --- admitted. However it is work that every designer of an API needs to do anyway. I really believe that some input from the ODF API projects like Oracles’ ODFDOM, IBM’s Simple API for ODF, ANR’s LPOD and Jos’ ODFKit could really help.
Let me finish my post by a classification of change to an ODF document:

I believe that for change tracking we only need “atomic operations” and a way to combine them to “compound operations”. I don’t think we need to be able to track changes to the XML tree or the XML text. In fact I think it does more harm than good.

2011-01-11T01:35:00.013+01:00

A lot has happened since my last blogpost in June 2009.

Its 2011 and I have been working for more than a year on a new project called “Native OfficeOpenXML” (NOOXML). The story is quite simple: I was very disappointed with the quality of the support of the “docx” format in OpenOffice.org. Even more --- I'm very disappointed with the code quality and the design! of the OpenOffice.org Writer core and layout. There are people who believe this can be solved by “code refactoring” fixing “low-hanging-fruits”, “quick wins” and other magic silver-bullet-phrases. But one thing was for certain: There is no way to (re-)implement a core and a layout engine. Can't be done. Impossible. No way.

OpenOffice.org took the refactoring route. I took the rewrite route.

After one year here is where we are.

What has happened:
I started designing and implementing the NOOXML-core in Jan 2010. The magic is the datastructure which allows a compact representation of the documents and fast implementation of insert/deletion operations etc. I also wanted to be able to do real- time-collaboration, which influenced the design of the core a lot. In March 2010 I was able to load the ECMA Spec Part I (very big document) into the core. Not only on a desktop machine, but also on my “iPod” (not “iPad”!!).
Once I had the basic core design and implementation done I started working on the layout engine. The primary goal was to build a fast and reliable layout engine. In my implementation I focused on OfficeOpenXML fidelity. In August I had the basic layout features like text, headers, footers, tables, footnotes etc. done. I was able to render the ECMA Spec Part I (again: very big document; >5000 pages) to PDF. I then added section and multiple column support.
Yesterday I was able to render the ECMA Spec Part I document on the iPod (real device) AND in the Android emulator (since I don't have an Android device) and without a user interface:

(I know: I took a really long time. But there is sooooo much room for improvements. And hey: OOo can't even load it on a desktop-machine.)

And here is the UI-less port for Android 2.3:

Happy new year!

2009-06-30T13:14:00.004+02:00

Bulk conversion

Before continuing the “API saga” I needed to have an infrastructure to be able to load a bulk of documents and save them using a certain filter. For me the reason was mainly for testing purposes, however its very convenient for “bulk conversion” too.
The syntax is:


./soffice.bin -bulk [targetDir]/[filterName].[targetExt] [dir] ... [dir]

E.g. the following call will convert all *.odt documents from /home/freuter/tmp/ to “/home/freuter/tmp/out/*.doc” documents using the “MS Word 97” filter:


./soffice.bin -bulk "/home/freuter/tmp/out/MS Word 97.doc" /home/freuter/tmp/*.odt

This command will convert all ~/tmp/*.doc documents to ~/tmp/out/*.odt using the ODF converter:


./soffice.bin -bulk ~/tmp/out/writer8.odt ~/tmp/*.doc

And finally this call will convert all ~/tmp/*.doc document to ~/tmp/out/*.pdf PDF document using the “ writer_pdf_Export” filter:


./soffice.bin -bulk ~/tmp/out/writer_pdf_Export.pdf ~/tmp/*.doc

The patch is here. I additionally fixed a bug in the m_nRequestCount logic and I enabled it in the [Experimental] section.

2009-06-26T09:18:00.004+02:00

API Design Matters

I was reading a very interesting article called "API Design Matters" with the subtitle "Bad application programming interfaces plague software engineering. How can we get things right?". Very cool stuff.

In OpenOffice.org we have an API "plague" too: The ODF import/export is based on the "UNO-API" and so is the OOXML import for Writer. And developers hate these APIs.

So the question is why do developers hate the "UNO-API"? And the obvious --- but wrong answer --- is: "I hate the UNO-API because of UNO". Don't get me wrong here: This is neither about pro or contra UNO. But the statement that "UNO is the problem of the ODF import/export and OOXML import problems" is wrong. It's not UNO per se, but its the design of the API.
[In case you're wondering what "UNO" is: UNO=COM ;-) So UNO is OpenOffice.org's way of COM.]

And just to be sure I do not offend the wrong people: The UNO-API was not designed to be used in the import/export filters. It was designed to be the API for "OpenOffice.org BASIC" developers, i.e. it was designed to provide a similar API to what VBA developers have in Microsoft Office. It was never designed to be used for import/export filters.

The problem was the decision to base the import/export code on such a high-level API! And we suffer from this decision until now!

Anyway. How can we fix this?
a) We claim the current API is the best mankind can do and print T-Shirts with 1000 years of OOo experience.
b) We claim UNO and abstraction is the problem and use the internal legacy APIs, so that we never get a chance to refactor the internal legacy stuff since we're creating even more dependencies.
c) We come up with a better API.

Option a) was demonstrated at the OpenOffice.org conference in Beijing. [Does anybody have a picture of the T-Shirt?]

Option b) is the straightforward approach. E.g. in Writer the “.DOC”, “.RTF”, .”HTML” filters are based on the internal “Core” APIs. So lets use these APIs instead of the UNO-APIs.
Whats wrong with the approach? The problem is that these internal APIs do not abstract from the underlying implementation at all. Repeat: The internal APIs do not abstract from the underlying implementation at all.
Does this answer the question why using the internal APIs is the wrong approach? Obviously *not* having an abstraction between your core implementation details and your import/exports filters is ... [offensive language detected ;-)].

Option c) only has one problem: How should the API look like?

I have some ideas here, but before posting them maybe there are some strong believes out there?

2008-04-15T18:02:00.005+02:00

Finally we had a developer conference! The good thing is that it was real fun. The bad thing was that I learned and drank toooooo much....

There are some dicussions I'd love to share with you:

Bug handling. Had some interresting chats about bug handling, responsiveness etc. from a developers point of view. Especially from a filter developers point of view. My believe is that we need a better clustering of bugs into problematic areas. This definetly will help to manage espectations as well as quality.
Mail merge. Learned that mail merge is not only broken IMHO but also in the opinion of others. Good (or bad ?:-)). However great things will happen here.
UI. Very good ideas about how to change the UI. Thanks Ricardo that was a great session.
Interop brokeness. Discussed my ideas about how to change ODF and OOo for better interop. Always good to get your ideas “blessed” by the master himself. Thanks Caolan...
Some chats about what to do with http://www.go-oo.org and how to attract more developer. Wait until my VM will appear... ;-)

Beside from the above some interresting news regarding OOXML/ODF/ISO arose. The report from the ISO meeting in Oslo sounds very promising IMHO:


<quote>
SC 34 envisages the creation of three distinct working groups that meet the needs of:
 1. ISO/IEC 29500
 2. ISO/IEC 26300
 3. Work on interoperability/harmonization between document format standards
    and wishes to incorporate existing expertise on these standards.
</quote;>

Only trouble here is that the ODF people do *not* seem to be happy about that --- but I have no idea why?

Overall it was a great week:

~Florian

2008-02-05T14:36:00.000+01:00

"XML Namespaces are designed to support exactly this kind of thing." (Tim Bray)

We make really good progress on our interoperability work. In our current focus area of fields we extended the OpenOffice.org Writer core for better support of MS Word-like fields. The first feature which benefits from this work are “Input fields” which now support the long wanted "tabbing" feature.

However we want all fields to benefit from the new enhanced field core --- not only "Input fields". Other areas are e.g "Mail merge fields" etc.. Since all of this fields share the same generic mechanism we decided to add support for this generic MS Word-like fields in OpenOffice.org Writer. But by doing so we faced the problem that ODF is not supporting these kind of fields.

Interestingly Tim Bray (Director of Web Technologies at Sun Microsystems) suggested a solution already in November 2005: http://www.tbray.org/ongoing/When/200x/2005/11/27/Office-XML. Unsurprisingly he suggested XML namespaces to solve this problem.

Thats what we did. MS Word-like fields are now stored in the namespace

xmlns:field="urn:openoffice:names:experimental:ooxml-odf-interop:xmlns:field:1.0"

which clearly indicates the purpose: OOXML<->ODF interoperability.

The following RelaxNG fragment enhanced the current ODF specification with the new fields:

<define name="paragraph-content" combine="choice">
 <choice>
  <element name="field:fieldmark">
   <attribute name="text:name">
    <ref name="string"/>
   </attribute>
   <attribute name="field:type">
    <ref name="namespacedToken"/>
   </attribute>
   <attribute name="field:locked">
    <ref name="boolean"/>
   </attribute>
         <sequence>
        <ref name="fieldmark-parameter"/>
        <zeroOrMore>
       <ref name="paragraph-content"/>
        </zeroOrMore>  
         <sequence>
     </element>
  <element name="field:fieldmark-start">
   <attribute name="text:name">
    <ref name="string"/>
   </attribute>
   <attribute name="field:type">
    <ref name="namespacedToken"/>
   </attribute>
   <attribute name="field:locked">
    <ref name="boolean"/>
   </attribute>
      <ref name="fieldmark-parameter"/>
  </element>
  <element name="text:fieldmark-end">
  </element>
 </choice>
</define>

In general fieldmarks are very similar to bookmarks, except that they need to be properly nested. This is achieved by the fact, that a field:fieldmark-end does not have a "name" attribute, but instead closes the last opened field:fieldmark-start element.
The field:fieldmark element is a short form of field:fieldmark-start and field:fieldmark-end. It SHOULD preferably be written instead of start-/end marks.

Every fieldmark can have

a name (text:name); similar to the name of text:bookmark elements. They SHOULD be unique. (Preferably also with the bookmark names).
a type (field:type) which allows application to define the type of the fieldmark.
a sequence of associated (name, value) pair represented by the <field:param field:name=”string” field:value=”string”/>.
a locked attribute which specifies whether the user can edit the content or not.

A sample. Lets take a loog at the following sample docs:

The OOXML representation is:

  <w:p>
   <w:r><w:t xml:space="preserve">Title: </w:t></w:r>
   <w:bookmarkStart w:id="0" w:name="Text1"/>
   <w:r>
     <w:fldChar w:fldCharType="begin">
       <w:ffData>
         <w:name w:val="Text1"/>
         <w:statusText w:type="text" w:val="Just a sample field."/>
         <w:textInput/>
       </w:ffData>
     </w:fldChar>
     <w:instrText xml:space="preserve"> FORMTEXT </w:instrText>
     <w:fldChar w:fldCharType="separate"/>
     <w:t xml:space="preserve">A sample input.</w:t>
     <w:fldChar w:fldCharType="end"/>
   </w:r>
   <w:bookmarkEnd w:id="0"/>
 </w:p>
 <w:p>
   <w:r><w:t xml:space="preserve">Description: </w:t></w:r>
   <w:bookmarkStart w:id="1" w:name="Text2"/>
   <w:r w:rsidR="00FA39C2">
     <w:fldChar w:fldCharType="begin">
       <w:ffData>
         <w:name w:val="Text2"/>
         <w:statusText w:type="text" w:val="Yet another sample field..."/>
         <w:textInput/>
       </w:ffData>
     </w:fldChar>
     <w:instrText xml:space="preserve"> FORMTEXT </w:instrText>
     <w:fldChar w:fldCharType="separate"/>
     <w:t>A sample input.</w:t>
   </w:r>
 </w:p>
 <w:p>
   <w:r><w:t>Second sample input paragraph.</w:t></w:r>
   <w:r><w:fldChar w:fldCharType="end"/></w:r>
   <w:bookmarkEnd w:id="1"/>
 </w:p>
 <w:bookmarkStart w:id="2" w:name="Check1"/>
 <w:p>
   <w:r>
     <w:fldChar w:fldCharType="begin">
       <w:ffData>
         <w:name w:val="Check1"/>
         <w:statusText w:type="text" w:val="A sample checkbox..."/>
         <w:checkBox>
           <w:checked/>
         </w:checkBox>
       </w:ffData>
     </w:fldChar>
     <w:instrText xml:space="preserve"> FORMCHECKBOX </w:instrText>
     <w:fldChar w:fldCharType="end"/>
   </w:r>
   <w:bookmarkEnd w:id="2"/>
   <w:r><w:t xml:space="preserve"> Make sense?</w:t></w:r>
 </w:p>

The ODF+Enhancement representation is:

 
   <text:p>Title: <field:fieldmark-start text:name="Text1" field:type="ecma.office-open-xml.field.FORMTEXT"><field:param field:name="Description" field:value="Just a sample field."/></field:fieldmark-start>A sample input.<field:fieldmark-end/></text:p>
   <text:p>Description: <field:fieldmark-start text:name="Text2" field:type="ecma.office-open-xml.field.FORMTEXT"><field:param field:name="Description" field:value="Yet another sample field..."/></field:fieldmark-start>A sample input.</text:p>
   <text:p>Second sample input paragraph.<field:fieldmark-end/></text:p>
   <text:p><field:fieldmark text:name="Check1" field:type="ecma.office-open-xml.field.FORMCHECKBOX"><field:param field:name="Description" field:value="A sample checkbox..."/><field:param field:name="Result" field:value="1"/></field:fieldmark><text:s/>Make sense?</text:p>

Cool isn't it. Or with Tim's words: "Who could possibly be against it?"

2008-01-23T23:30:00.000+01:00

Never try to catch a train last minute...

Yesterday I tried to catch a train last minute. While running towards it I fell down. I got up again and managed to get it.

While sitting in the train I realized that my arm hurts and at my destination I went into a hospital. The X-rays revealed that my ellbow was broken ;-) Nohting serious --- it'll hopefully heal within two weeks...

So my advice clearly is: Never try to catch a train last minute --- let it pass!

And the moral is: The next train would have departed in only 30 minutes...

Damn!

P.S. In the next two weeks you'll only get short messages from me since I can only type with one hand :-(

2007-12-18T16:13:00.000+01:00

Back to the binaries! Yeah!

After all this XML work the binary file formats are a different world. For the fields work I needed to analyze the “form field” structure of the binary .DOC format:

The header: Actually a misused PICT structure:

b10	b16	field	Type	size	bitfield	comments
0	0	lcb	U32			Count of bytes of the whole block.
4	4	cbHeader	U16			Always 0x44
6	6		U8[62]			Contains zero. In fact this is the PICT struct, but since its not need we can fill it with zeros.

The formfield payload (Unicode Variant)

b10	b16	Field	Type	size	bitfield	comments
0	0	cUnicodeMarker	U8[32]			Contains {0xFF,0xFF,0xFF,0xFF}
4	4	fftype	U8	:2	03	Type: 0 = Text 1 = Check Box 2 = List
		ffres	U8	:5	7C	Result field for a form field. Values from 0 to N-1, where N is the number of \ffl entries. In case of check boxes: 0==unchecked; 1==checked.
		ffownhelp	U8	:1	80	1 if there is associated Help text, 0 otherwise.
5	5	ffownstat	U8	:1	01	1 if there is associated status line text, 0 otherwise.
		ffprot	U8	:1	02	1 if this field is protected, 0 otherwise.
		ffsize	U8	:1	04	Type of size selected for check box field: 0 = Auto 1 = Exact
		fftypetxt	U8	:3	38	Type of text field: 0 = Regular text 1 = Number 2 = Date 3 = Current date 4 = Current time 5 = Calculation
		ffrecalc	U8	:1	40	1 if the field should be calculated on exit, 0 otherwise.
		ffhaslistbox	U8	:1	80	1 if this field has list box attached to it, 0 otherwise.
6	6	ffmaxlen	U16	:15	7FFF	Number of characters for text field. Zero means unlimited.
			U16	:1	8000	Unknown. Set to zero.
8	8	ffhps	U16			Check box size (half-point sizes).
10	A	xstz_ffname	Xstz_UString0			Form field name
		xstz_ffddeftext	Xstz_UString0			Default text for field. Only if type==0.
		ffdefres	U16			Default resource for list field. Default value for check box (0=default unchecked; 1=default checked). Only if type!=0.
		xstz_ffformat	Xstz_UString0			Format for text field
		xstz_ffhelptext	Xstz_UString0			Help text
		xstz_ffstattext	Xstz_UString0			Status line text
		xstz_ffentrymcr	Xstz_UString0			Macro to execute upon entry into this form field
		xstz_ffexitmcr	Xstz_UString0			Macro to execute upon exit from this form field
		cUnicodeMarker2	U8[2]			Contains {0xFF, 0xFF}; Padding and/or indicator for Unicode?
		fflLen	U32			Num of ffls
		ffl	Xstz_UString[fflLen]			Resource string for lists.

An Xstz_UString has the following form:

b10	B16	Field	type	size	bitfield	Comments
0	0	Len	U16			Len of the String.
2	2	Unicode char	U16[len]			Unicode chars

An Xstz_UString0 has the following form:

b10	B16	Field	type	size	bitfield	Comments
0	0	len	U16			Len of the String.
2	2	Unicode char	U16[len]			Unicode chars
2+2*len		Zero	U16			Trailing “0”

In case of non-Unicode encoding then the Unicode Marker disappear and the string chars have U8 size.

You might also want to take a look at the ffData element in OOXML ;-)

2007-10-30T23:40:00.000+01:00

Business applications of unstructured text

Interresting article in the ACM Communications.

A widely touted IT factoid states that
80% of the information produced by
and contained in most organizations
is stored in the form of unstructured
data. Most of it is text (such as memoranda,
internal documents, email,
organizational Web pages, and comments
from customers and from
internal service personnel), and most
of the applications that reflect the
value of unstructured data are able to
process it. Although unstructured
data takes other forms, including
images and audio, here I focus on the
applications, technologies, and architectures
for unstructured text acquisition
and analysis (UTAA).

2007-10-29T21:51:00.001+01:00

New OpenOffice.org target.

Many of you probaly know the “WONT FIX” target in the OpenOffice.org issue tracker.

What about introducing a new target: “HELPS MICROSOFT”.

But why do we need this? These days many people --- especially from the file formats camps --- are extremely sensitive of anything related to compatiblity 'cause they believe it helps Microsoft.

So lets give the ODF warriors an opportinity to clearly communicate with the users. Give them the “HELPS MICROSOFT” target to publicly exposing the issuer of the bug and the people working on it.

2007-10-25T14:14:00.000+02:00

Field update --- preview for Windows.

I now have a preview for Windows available at http://download.go-oo.org/preview/oodemo.zip.

Simply download it and unzip it. To start execute soffice.exe in ooo2.3/program/.

Same features as the Linux Version. So no saving at this point.

And don't forget to give feedback :-)

Thanks,

~Florian

2007-10-24T17:24:00.000+02:00

IBM's Symphony.

Downloaded IBM's Symphony today to follow up on some of the problems discussed at the ODF Interop Camp. (Btw. its sad that the ODF Camp people want to treat the problems as confidential.).

So back to Symphony. Why the hell did they crippled all the cool OpenOffice.org easter eggs?

So why is =game("StarWars") crippled?

And look what they done to the lovely picture of the Calc team:

I think that contradicts the SISSL :-)

2007-10-15T21:10:00.000+02:00

Update on field work --- Early preview available for Linux.

In a previous post I talked about my field-proof-of-concept. I continued to work on the issue and I'm happy to give an update on that front.

You can download a preview version of my work here:

http://download.go-oo.org/preview/oodemo.tgz
(Linux only. Just untar the archive tar xzf oodemo.tgz and then cd oodemo/program and start ./soffice). This is a preview version. Do not use it for productive work! The preview demo shows

the core enhancements (tabbing!), and

.DOC import.

The work for .DOC export, ODF import/export is not done and not included in the demo.

For testing you can download the sample file formc1new.doc which is taken from issue 79720. It should look like this:

Again --- this is work in progress. So do not expect everything to work. However if you have issues please let me know. And remember “saving does not work yet :-)”.

I really hope I get some feedback,

~Florian

P.S.
I will make the patch available ASAP. It is the result of some weekend hacking --- it really needs some polishing first.

2007-09-23T20:51:00.000+02:00

Back from the OpenOffice.org Conference 2007 in Barcelona

Good to meet people in person.

Talked at lot about

Harmonization between ODF and OOXML,
Trade-off between Standardization and Innovation and
Interoperability wrt. ODF<->ODF and ODT<->.DOC

Some pics from the conference:

(Thanks Peter for great evening.)

(Thanks to the people at the ODF Camp)

(Thanks to Kohei, Hubert and Noel for the great time at the XXVII MOSTRADA DE VINS I CAVES DE CATALUNYA)

I hope I can find some time to go into more details.

2007-09-12T16:00:00.000+02:00

Office 2.0 conference

I'm just back from the Office 2.0 conference in San Francisco. I was participating in a panel discussion about Document Formats:

Very nice crowd the Office 2.0 guys. They really taught me to think more about collaboration.

Many thanks for the nice presentations. (And for the insight that most of you use OpenOffice.org to convert between HTML and the other file formats ;-))

Ahh -- and almost forget. I will look into OpenSAM. Promised. Sounds really like a good idea for OpenOffice.org supporting it.

2007-08-13T11:23:00.001+02:00

Status of my Suggested enhancements for OpenDocument V1.2:

Hi Thomas,

thanks for the question. Here is the status:

Tables:
* introduce allowCollapse attribute for paragraphs following nested tables to encode WW and HTML-like tables.	Not put up for discussion.
* declare sub tables as deprecated	Under discussion in the Accessibility SC.
Numbering
* introduce text:level-text attribute to encode arbitrary number formats	Rejected.
* introduce text:num-follow-char to encode WW-like numbering	Partly accepted.
* introduce text:list-override to encode WW-like numbering	Strongly rejected.
* declare style:list-level-properties/@text:space-before as deprecated. Effect can be achieved with paragraph indent.	Rejected.
Master-page styles
* add header-first and footer-first to encode WW-like page-styles	Not put up for discussion
* modify master-page styles such that WW-like sections can be encoded; current CSS3.0 like text:sections are not applicable	Not put up for discussion
* declare the style:next-style-name attribute of master-page declarations as deprecated.	Not put up for discussion
Styles:
* allow deriving paragraph-family styles from text-family styles.	Not put up for discussion
"Break chars"
* introduce a command and a command similar to the command	Not put up for discussion
Fields:
* enhance field support by introducing a <text:field-start/> and a <text:field-end/> element to which metadata can be attached.	Rejected
Change tracking:
* introduce change tracking for tables	Not put up for discussion
* introduce change tracking on property level	Not put up for discussion
Discourage the use of the following OD features for MOOX interop:
* nested frames	Not put up for discussion / Internally communicated as rejected.
* current CSS3.0 like text:sections	Not put up for discussion / Internally communicated as rejected.
* use fo:break-before instead of fo:break-after	Not put up for discussion / Internally communicated as rejected.
* use fo:margin-* for tables	Not put up for discussion / Internally communicated as rejected.

In general I must confess the OpenDocument TC didn't picked up my discsussion topics... (It's listed as suggested but never has been put for discussion into the agenda). Additionally I had a lot of private communiation where my ideas where communicated as unwanted/rejected.

To get an idea of whats discussed for ODF1.2 take a look at:

Proposals under discussion

Proposals for consideration for a vote in the next coordination call

Approved Proposals

Proposal integrated into the specification document

2007-07-18T14:22:00.001+02:00

Field enhancement proof-of-concept finished.

I've been working on field enhancement for OpenOffice.org Writer for quite a while and today I finished my proof-of-concept hacking:

OpenOffice.org Writer has a lot of shortcommings wrt. to fields which I tried to address:

Allow for in-place editing of input field (turn off pop-up)

allow use of different fonts within one field

Nested Fields support on Write and allow nested conditions when using conditional fields

In my proof-of-concept I was able to enhance the Writer core such that these issues are addressed. (That's the good news!)

Unfortionately my proof-of-concept still needs a lot of love. First thing is to clean up the prototype and generate patches for ooo-build.

However I'm happy since this is my first major work on the OpenOffice.org Writer layout and the field support is an issue in OpenOffice.org Writer for quite a while...

2007-07-13T12:40:00.000+02:00

"The first casualty of War is Truth"
Reading some blogs about the ODF/OOXML file format war the famous quote "The first casualty of War is Truth" (from Rudyard Kipling --- I guess) comes into my mind.

2007-06-28T16:53:00.000+02:00

XEMBED, Mono and OpenOffice.org

In my last post I talked about the hack I did to get some Java applets running in an OpenOffice.org docking window.

I played a little more with the code and managed to get a XEMBED socket running in an OpenOffice.org docking window. The picture below shows an OpenOffice.org running with a XEMBED ready docking window:

The title bar of the docking window shows the socket id to which XEMBED applications can connect.

I used the following Mono code to connect to the XEMBED socket:


using System;
using Gtk;

// Compile with:
// mcs -pkg:gtk-sharp SamplePlug.cs

public class SamplePlug
{

 public static void Main(string[] args) {
     if (args.Length != 1) {
  Console.WriteLine("Need socket id as an argument.");
  return;
     }
     uint socket_id = UInt32.Parse(args[0]);
     
     Console.WriteLine("using socket "+socket_id);
     
     Application.Init();
     
     Plug plug= new Plug(socket_id);
//     plug.Add(new Label("HELLO"));
     plug.Add(new Entry("HELLO"));
     plug.ShowAll();

     Console.WriteLine("running..");
     Application.Run();
 } 
}

The picture below shows it all running:

In theory this'll work not only with Mono but with any application which can talk the XEMBED protocol like e.g. GTK- and QT-based applications.

2007-06-22T15:38:00.000+02:00

ActiveX-like embedding of applications in OpenOffice.org?

I'm working on the problem of embedding applications in an OpenOffice docking window --- similar to the new API inside Word which allows to embed ActiveX applications in a task pane.

I just started “hacking” on the problem and I decided to try whether I can embed a Java Applet inside an OpenOffice.org docking window.

Well I “hacked” :-) the Navigator --- which is a docking window --- and reused some code from the sj module and here it is:

However there a some --- severe --- open problems:
- Focus: The framework knows nothing about XEMBED_REQUEST_FOCUS, etc
(http://standards.freedesktop.org/xembed-spec/latest/ar01s05.html)
So there is a focus problem when the applet is running in "docked" mode. (Maybe this is also the problem why the OOoBean is not working so good?)
- Need to clone the Navigator code...
- Resize problem and other events. Currently the embedded app is not notified
about resize events.
- Need to define an API to get link between embedded aps and the custom
pane.
- Some "solar mutex" problems as always :-)

2007-04-24T15:34:00.000+02:00

Smart Documents.

Yesterday I started the evaluation of the Microsoft's ISmartDocument interface for Office2003.
I started in a clean VM. I installed:
* Windows XP
* Office2003 and
* DevEnv 7.1 (C#)

Then I downloaded and installed the Office 2003 Smart Document Software Development Kit (SDK)

Next I tried to compile the “SimpleSampleCS” from the SDK.

It failed.

Some research showed that I had also to install the Office 2003 Update: Redistributable Primary Interop Assemblies.

Finally I was able to compile the project.

Next I tried to activate the smart document by simply opening the “SimpleSample.doc” in Word 2003. I ran the “setpolicy.bat” script and then the “DisableManifestSecurityCheck.reg” file. Then I opened the “SimpleSample.doc” and was asked to “Download the XML expansion pack”. After answering the security question with “no” I was able to get the sample working. (Other orders of invoking the scripts lead to the failure of loading the expansion pack”. I took me quite a while to figure that out. I even installed the .NET 3.0 Runtime Environment --- but I believe that is not required.).

Finally I was able to see the following “SimpleSampleCS”:

The overall architecture of “ISmartDocument”s is based on custom schemata to which actions can be attached. Quite simple but I gives custom schemata a meaning of the end.

The question I'm interested in:
* How is this work related to alien attributes and metadata in ODF? And
* what do we have to do to provide a competitive environment in OOo and ODF?

2007-01-08T16:22:00.000+01:00

WARNING: THIS BLOG IS TOO LONG AND TOO TECHNICAL :-)

How to improve interoperability: A case study with sections

Q: So --- OpenOffice.org Writer has section support. Microsoft Word has section support. Where's the problem?
A: In the details.

OpenOffice.org section

The OpenOffice.org sections are very similar to HTML MULTICOL resp. CSS3 module: Multi-column layout concepts where a sequence of paragraph level content like paragraphs, tables, etc. can be grouped together in a <text:section> and multiple columns can be requested for layout.

Sections in a Writer document have the following form:


WRITERDOC ::= (PARAGRAPH | TABLE | WRITERSECTION)+
WRITERSECTION::= <text:section> (PARAGRAPH | TABLE  | WRITERSECTION)+ </text:section>
TABLE ::= TABLEROW+
TABLEROW ::= TABLECELL+
TABLECELL ::= (PARAGRAPH | TABLE | WRITERSECTION)+

So WRITERSECTIONs in OpenOffice.org can start and end anywhere and can be nested.

Microsoft Word sections

Microsoft Word sections are different. Conceptually every Microsoft Word document consists of at least one section. So Microsoft Word documents have the form


WORDDOC ::= WORDSECTION+
WORDSECTION ::= (PARAGRAPH | TABLE)* PARAGRAPH[ with section properties attached to it]

Please note that Microsoft Word sections are always “top-level” and that only a paragraph can trigger a new Microsoft Word section to start after the itself.

So what does this mean for conversion?

a) Every WORDOC can be mapped to a WRITERDOC [not quite true for other reasons, but lets forget about this detail :-)];
b) *NOT* every WRITERDOC can be mapped to a WORDDOC.

Consider the following WRITERDOC:


PARAGRAPH
<text:section>
PARAGRAPH
<text:section>
PARAGRAPH
PARAGRAPH
</text:section>
PARAGRAPH
</text:section>
PARAGRAPH

The above WRITERDOC can not be mapped to a WORDDOC. OpenOffice.org will change the structure on export and write a WORDDOC like


PARAGRAPH + section props
PARAGRAPH + section props
PARAGRAPH 
PARAGRAPH  + section props
PARAGRAPH  + section props
PARAGRAPH

When importing the WORDDOC back into OpenOffice.org Writer the WRITERDOC will look like


<text:section>
PARAGRAPH
</text:section><text:section>
PARAGRAPH
</text:section><text:section>
PARAGRAPH
PARAGRAPH
</text:section><text:section>
PARAGRAPH
</text:section><text:section>
PARAGRAPH
</text:section>

So clearly the structure has changes and roundtrip is broken. You can generated an infinite number of roundtrip problems based on this.

The stuff can get even worse. Consider the following WRITERDOC:


<text:section>
PARAGRAPH
<TABLE>
...
</TABLE>
</text:section>
PARAGRAPH

In order to export this to a WORDDOC you need to add a new paragraph:


PARAGRAPH
<TABLE>
...
</TABLE>
PARAGRAPH + section break properties
PARAGRAPH

since you can only tell a PARAGRAPH to start a new section.

BUT... why are we trying to map WORDSECTIONs to WRITERSECTIONS?

I believe that its better to map between WORDSECTIONs and WRITERMASTERPAGESECTIONs.
Let me try to explain.

In OpenOffice.org Writer you have the concept of “page styles”. When rewriting the above grammar for WRITERDOCs including page styles we get


WRITERDOC ::= WRITERMASTERPAGESECTION+
WRITERMASTERPAGESECTION ::= (PARAGRAPH+master page break before | TABLE + master page break before) (PARAGRAPH | TABLE )*

which is quite similar to


WORDDOC ::= WORDSECTION+
WORDSECTION ::= (PARAGRAPH | TABLE)* PARAGRAPH[ with section properties attached to it]

right? So a “master page break before” attribute can be put to a paragraph or a table and causes the new WRITERMASTERPAGESECTION to start before the paragraph or table. Whereas a “section property “ on a paragraph causes the start of a new WORDSECTION after the paragraph in a WORDDOC. Such a WORDSECTION can start on a new page with new header/footer or be continuous and simply change the columns settings for the following content.

So my favorite idea is to allow a “master page override” property at a paragraph in a WRITERDOC and use this new property to handle WORDSECTIONS.

More concrete I would like to add the following attributes to a WRITERDOC paragraph or table style:


<define name="style-style-attlist" combine="interleave">
 <optional>
  <attribute name="style:master-page-override">
   <ref name="styleNameRef"/>
  </attribute>
  <attribute name="style:master-page-break">
   <choice>
    <value>auto</value>
    <value>column</value>
    <value>page</value>
    </choice>  
  </attribute>
 </optional>
</define>

we could then use this attributes to handle WORDSECTION 100% (and discourage the use of WRITERSECTIONS for .DOC interoperability :-))

For backward compatibility also the style:master-page-name="N" attribute could be emitted in case of style:master-page-override="N" and style:master-page-break="page".

E.g. the OfficeOpenXML fragment


<w:body>
 <w:p>A1</w:p>
 <w:p>A2</w:p>
 <w:p>A3<w:pPr><w:sectPr><w:cols w:num="3"></w:sectPr></w:pPr></w:p>
 <w:p>B1</w:p>
 <w:p>B2<w:pPr><w:sectPr><w:cols w:num="2"><w:type w:val="continuous" /></w:sectPr></w:pPr></w:p>
 <w:p>C1</w:p>
 <w:p>C2</w:p>
 <w:pPr><w:sectPr><w:cols w:num="1"></w:sectPr></w:pPr>
</w:body>

could be translated to the OpenDocument fragment


<office:automatic-styles>
<style:style name="S1" style:family="paragraph" style:master-page-override="P1" style:master-page-break="page"/>
<style:style name="S2" style:family="paragraph" style:master-page-override="P2" style:master-page-break="auto"/>
<style:style name="S3" style:family="paragraph" style:master-page-override="P3" style:master-page-break="page"/>
..
<style:page-layout style:name="PL1">
 <style:page-layout-properties>
  <style:columns fo:column-count="3"/>
 </style:page-layout-properties>
</style:page-layout>
<style:page-layout style:name="PL2">
 <style:page-layout-properties>
  <style:columns fo:column-count="2"/>
 </style:page-layout-properties>
</style:page-layout>
<style:page-layout style:name="PL3">
 <style:page-layout-properties>
  <style:columns fo:column-count="1"/>
 </style:page-layout-properties>
</style:page-layout>
</office:automatic-styles>
..
<office:master-styles>
 <style:master-page style:name="P1" style:page-layout-name="PL1"/>
 <style:master-page style:name="P2" style:page-layout-name="PL2"/>
 <style:master-page style:name="P3" style:page-layout-name="PL3"/>
</office:master-styles>

..
<office:body>
<text:p text:style-name="S1">A1</text:p>
<text:p>A2</text:p>
<text:p>A3</text:p>
<text:p text:style-name="S2">B1</text:p>
<text:p>B2</text:p>
<text:p text:style-name="S3">C1</text:p>
<text:p>C2</text:p>
</office:body>

and back!

I guess this blog entry is far to long and technical by now. Maybe I should move to a WIKI in the future.

~Florian

2006-12-11T23:40:00.000+01:00

Successfull deployment of OpenOffice in the public sector.

I had the opportunity to participate at the MAMPU conference on Successfull deployment of OpenOffice in the public sector in Kuala Lumpur, Malaysia.

It was great.

The general arguments I tried to make in my talk where around the following matrix:

The argument I derived from that matrix where

clearly a shift from an open source application with a proprietary file format to an open source application with an open standardized file format improves openess.
however how can a shift from a proprietary application with a proprietary file format to a proprietary application with an “open” file format improved openess? And --- is it the same kind of “openess” as in case 1)?

I then made the argument that only an open source application with an open file format can prevent a “lock-in” situation. I illustrated the fact by a case study of supporting a “minority language”.

So --- how would you fill out the matrix?

2006-11-27T11:08:00.000+01:00

Note: This element should not be used in new documents.

I still got a lot feedback regarding the use of the word „deprecated“. So please let me explain what I had in mind when I wrote this.

In the current ODF specification there is the following note in section 6.6.11 Table Formula Field. It says:

Note: This element should not be used in new documents.

My idea was simply to add a similar note to e.g. section 8.2.6 Subtables.

So again there are already features in ODF marked as "not to use".

2006-11-23T16:58:00.000+01:00

Hi Mathias,

thanks so much for referring to my blog and commenting on my "ODF 1.2 suggestions".

Yes --- I really believe we can work together making OpenDocument as well as OpenOffice.org better.

Regarding “deprecation”. Deprecation does not mean removal or even making OpenOffice.org worse. Never intentend to say this. Sorry for the confusion. What I meant was e.g. in the case of “sub tables” that "deprecation" or "discourating" the use of "sub tables" just means helping e.g. blind people to better use OpenDocument and OpenOffice.org: Subtable Accessibility Issue.

Of course the feature will stay, but I think we should offer the user an alternative way to achieve the same result and help --- not only accessibility --- but also interoperability.
Luckily OpenDocument has this feature already by using “row spans”.

I hope I could also clarify Michael Brauers concerns on the the OpenDocument TC mailing list with my response.

But I think we have the same idea here. Will never again use the word "deprecation" or "discourage" again. And I will not take away any feature. Promised!

~Florian

2006-11-20T17:08:00.000+01:00

Suggested enhancement for OpenDocument V1.2

Tables:
* introduce allowCollapse attribute for paragraphs following nested tables to encode WW and HTML-like tables.
* declare sub tables as deprecated

Numbering
* introduce text:level-text attribute to encode arbitrary number formats
* introduce text:num-follow-char to encode WW-like numbering
* introduce text:list-override to encode WW-like numbering
* declare style:list-level-properties/@text:space-before as deprecated. Effect can be achieved with paragraph indent.

Master-page styles
* add header-first and footer-first to encode WW-like page-styles
* modify master-page styles such that WW-like sections can be encoded; current CSS3.0 like text:sections are not applicable
* declare the style:next-style-name attribute of master-page declarations as deprecated.

Styles:
* allow deriving paragraph-family styles from text-family styles.

“Break chars”
* introduce a <text:page-break/> command and a <text:column-break/> command similar to the <text:line-break/> command

Fields:
* enhance field support by introducing a <text:field-start/> and a <text:field-end/> element to which metadata can be attached.

Change tracking:
* introduce change tracking for tables
* introduce change tracking on property level

Discourage the use of the following OD features for MOOX interop:
* nested frames
* current CSS3.0 like text:sections
* use fo:break-before instead of fo:break-after
* use fo:margin-* for tables

Unfortunately I think this list is not complete yet:-)

Feedback appreciated.

2006-11-15T22:23:00.000+01:00

All the small things…

…are really important in document conversion. The table below shows three different ODF features (numbered lists, nested styles as well as nested tables) and the way OpenOffice.org Writer resp. the CleverAge ODF Converter exports these.

Original ODF in OO.o	OO.o export to .DOC	CleverAge ODF Converter

The above examples look rather artificial. However these kinds of small problems can tell much about the conversion quality of a filter.
For example handling nested styles correctly is a huge effort for a rather small output. Thus everybody ignores them in the first shot and aims for the more visible features. You simply file a bug and set it to “later”.
The problem is when “later” comes you realize that handling nested styles requires major changes in your existing code. You then realize that the risk of loosing all your existing features by implementing nested styles is high. So “later” becomes a synonym for “never”.

In my opinion a good test for the underlying code quality of a conversion filter is to test the nasty stuff. I believe it can tell you a lot about whether “later” means “never” or simply “not yet”.

Btw. all the above features can be mapped correctly.

2006-11-14T23:53:00.000+01:00

So I started blogging. This is the first entry of my personal blog.

So there is a lot to clarify. But first let me provide some background to those who don’t know me.

I’ve done some work in the past on the .DOC, .RTF and WordML filters in OpenOffice.org. I also did some work on OpenDocument in the OpenDocument TC and the OpenDocument Metadata SC.

I also worked together with the OpenDocument Foundation on the ODF plug-in for Microsoft Office. The ODF plug-in for Microsoft Office is based on a library called OpenDocument Infoset API I was working on. Of course I’ll continue my voluntary work as a CTO of the OpenDocument Foundation and I’m eager to see how the Foundation’s work --- especially on the OpenDocument InfoSet API --- will continue. I’m happy to tell more about this in following blog entries if there is any interest.

What I discovered in all my work regarding the fileformats is that interoperability can be improved. In fact minor changes in the fileformats resp. implementing applications can improve interoperability significantly. And I hope I can prove this with my future work.

So what is this future work?

One very interesting area is OpenDocument Version 1.2. Currently the OpenDocument TC is working on the version 1.2 of OpenDocument which targets better interoperability. Very interesting and with great potential.

Another interesting area is OpenOffice.org. I really believe we can improve OpenOffice.org’s interoperability by implementing OpenDocument Version 1.2 in OpenOffice. This is a community effort for sure, so I hope I can attract people participating in this effort.

Maybe the next thing I should do is to have a little tutorial about how to contribute to the OpenOffice.org filters? Good idea?