June 15, 2013
This week is full of potentially incompatible changes. Excluding first change I’ve mentioned before, they are rather minor. Still, they could break code depending on them.
Incomatible changes
Rakudo Perl
Niecza Perl
New features
Perl 6 specification
-
Documented IO.copy, IO.rmdir, and IO.unlink.
These work just like UNIX system calls, except are less lowlevel (fail, instead of simply returning error codes). The exception is IO.copy that also has :$createonly argument that causes it to fail when file already exists.
-
dir and methods documented above now soft fail.
Because some may not understand, especially having experience with or die idiom from Perl 5 (this commit has a commit), I’m going to explain. The fail doesn’t mean returning error code. It means returning special sort of value. If the value won’t be specifically checked (the function is used sink context, or without boolean check), the object will explode.
For example, following snippet of code will fail when file won’t exist.
unlink "file";
But this won’t, as I make boolean check.
unlink "file" or $*ERR.say: "Oh noes.";
-
try implies use fatal.
Soft errors are considering fatal in try blocks.
-
constants, enums, and our variables are now exportable.
STD.pm6
Rakudo Perl
-
Implemented :$<foo> syntax in regexes.
-
Removed combination of :exists and :k.
Have I mentioned it previous week? Sorry, it was a printing error (assuming you can make a printing error on the web). It doesn’t actually make sense. It was removed one day just after I made previous article.
-
Fixed error for my &foo; multi foo() { }.
Previously it returned “Method ‘is_dispatcher’ not found for invocant of class ‘Any’“. It was LTA (Less Than Awesome), and was corrected.
-
Comma operator is disallowed in reduce.
All it was causing were infinite loops for lists containing more than one element. As it was rather pointless, it’s now disallowed. If you really want it, provide a code block.
-
perl6 -v returns version under Windows.
Previously, configuration script was calling git describe --match '2*'. The problem is that Windows doesn’t support single quotes. It was changed to double quotes to avoid problems.
-
Converting NaN or Inf to Rat returns itself.
NaN is so magical that it actually ignores cast requests. The previous behavior of casting NaN to 0, and erroring on Inf was just wrong.
-
{}:kv, {}:p, {}:k, and {}:v work just like in spec.
-
$*CWD and $*TMPDIR are now IO::Path objects.
-
Hash[TVal,Tkey].new works.
.new now looks into Hash type constraints while making new Hash.
-
Use of undef refers to Any instead of Mu.
-
Hash shapes have better error messages.
-
$default doesn’t leak to GLOBAL::.
-
Implemented Hash.classify, and Hash.categorize.
-
Some list methods fail early on infinite list.
As stuff like picking random element from infinite list doesn’t make sense, it makes sense to fail fast.
Niecza Perl
June 15, 2013 07:00
June 08, 2013
The JVM support of Rakudo Perl is mostly usable now (it’s still very buggy, but you can run “Hello, world!”). The instructions to install Rakudo on JVM are available in README file.
Incompatible changes]
Perl 6 specification
-
Removed tcuc.
Forcing the first character to titlecase, and rest of the characters to uppercase wasn’t the most useful thing. It was rather pointless. If you really deal with Unicode edge cases, use .uc.tc instead.
Rakudo Perl
Niecza Perl
New features
Perl 6 specification
STD.pm6
Rakudo Perl
Niecza Perl
June 08, 2013 07:00
June 03, 2013
[This is a response to the Russian Perl Podcast transcribed by Peter Rabbitson and discussed at blogs.perl.org.]
I found this translation and podcast to be interesting and useful, thanks to all who put it together.
Since there seems to have been some disappointment that Perl 6 developers didn’t join in the discussions about “Perl 7″ earlier this year, and in the podcast I’m specifically mentioned by name, I thought I’d go ahead and comment now and try to improve the record a bit.
While I can’t speak for the other Perl 6 developers, in my case I didn’t contribute to the discussion because nearly all the things I would’ve said were already being said better by others such as Larry, rjbs, mst, chromatic, etc. I think a “Perl 7″ rebrand is the wrong approach, for exactly the reasons they give.
A couple of statements in the podcast refer to “hurting the feelings of Perl 6 developers” as being a problem resulting from a rebrand to Perl 7. I greatly appreciate that people are concerned with the possible impact of a Perl 5 rebrand on Perl 6 developers and our progress. I believe that Perl 6′s success or failure at this point will have little to do with the fact that “6 is larger than 5″. I don’t find the basic notion of “Perl 7″ offensive or directly threatening to Perl 6.
But I fully agree with mst that “you can’t … have two successive numbers in two brands and not expect people to be confused.” We already have problems explaining “5″ and “6″ — adding more small integers to the explanation would just make an existing problem even worse, and wouldn’t do anything to address the fundamental problems Perl 6 was intended to resolve.
Since respected voices in the community were already saying the things I thought about the name “Perl 7″, I felt that adding my voice to that chorus could only be more distracting than helpful to the discussion. My involvement would inject speculations on the motivations of Perl 6 developers into what is properly a discussion about how to promote progress with Perl 5. I suspect that other Perl 6 developers independently arrived at similar conclusions and kept silent as well (Larry being a notable exception).
I’d also like to remark on a couple of @sharifulin’s comments in the podcast (acknowledging that the transcribed comments may be imprecise in the translation from Russian):
First, I’m absolutely not the “sole developer” of Perl 6 (13:23 in the podcast), or even the sole developer of Rakudo Perl 6. Frankly I think it’s hugely disrespectful to so flippantly ignore the contributions of others in the Perl 6 development community. Let’s put some actual facts into this discussion… in the past twelve months there have been over 6,500 commits from over 70 committers to the various Perl 6 related repositories (excluding module repositories), less than 4% (218) of those commits are from me. Take a look at the author lists from the Perl 6 commit logs and you may be a little surprised at some of the people you find listed there.
Second, there is not any sense in which I think that clicking ”Like” on a Facebook posting could be considered “admitting defeat” (13:39 in the podcast). For one, my “Like” was actually liking rjbs’ reply to mst’s proposal, as correctly noted in the footnotes (thanks Peter!).
But more importantly, I just don’t believe that Perl 5 and Perl 6 are in a battle that requires there to be a conquerer, a vanquished, or an admission of defeat.
Pm
June 03, 2013 20:13
June 01, 2013
This is relatively small, but big update. Small in term of number of commits, big in terms of new features. IO methods already implemented in implementations were added to specification. New array methods that avoid flattening are now in Synopsis. New features to help with bitwise operations, msb and lsb also were added - and implemented in Perl 6 implementation.
jnthn is working on new, lightweight virtual machine for Rakudo called MoarVM, intended to be optimized for Perl 6. It already supports 6model (as its core object system), Unicode strings, GC, IO, binary bytecode, and more. You can get more information about MoardVM at jnthn’s blog.
Incompatible changes
Perl 6 specification
-
Changed START to once.
START wasn’t a real phaser, because it was executing inline. As uppercase keywords are reserved for phasers that execute off-flow (for example, CATCH executes when exception happens in block where it’s contained, or BEGIN that executes at beginning).
Also, START was bad name for something that runs once.
STD.pm6
New features
Perl 6 specification
-
Added IO::Handle.encoding.
This method can be used to determine current file encoding. It was already implemented for a long time in current specification, but just got a specification.
-
Added IO::Handle.open.
The method can be used for reopening filehandles, or reusing them for different files. It was also already implemented in implementations, so it makes sense to spec it.
-
Added :r, :w, and :a parameters to open.
They stand for read, write, and append. Those already existed in implementations, and just like examples above, was added to specification.
-
Added IO::Handle.IO.
Casts IO::Handle to IO::Handle. It’s actually useful, in certain cases.
-
Added Int.msb and Int.lsb.
msb returns the position of most significant bit (generally log2 of the number rounded down, but without floating point issues). lsb returns the position of least significant bit that is 1.
-
Added .arg and .args methods.
By default, .[] and .elems flatten Parcels.
> ('first', ('second', 'third'), 'fourth')[2]
third
> ('first', ('second', 'third'), 'fourth').elems
4
Sometimes this behavior is unwated. In these cases you can use new methods, .arg and .args.
> ('first', ('second', 'third'), 'fourth').arg(2)
fourth
> ('first', ('second', 'third'), 'fourth').args
3
Rakudo Perl
Niecza Perl
June 01, 2013 07:00
May 31, 2013
Here's what I wrote in another post earlier this year:
[...] what makes me the most optimistic about the Perl 6 effort: after a few years of watching things evolve, I've noticed that while Perl 6 is being developed top-down on the outermost scale, it's actually a series of bottom-up projects that drive Perl 6 forwards.
Over the years, Perl 6 users have seen a descending sequence of components being replaced and improved, each one furthering our reach. It reads almost like a nursery rhyme, though with a positive spin:
- When we had junctions, we got ourselves a method dispatch that served them better.
- When we had method dispatch, we got ourselves an object system that served it better.
- When we had an object system, we got ourselves 6model that served it better.
And people have been talking about taking the obvious next step with this: building a runtime, or retrofitting an existing runtime, that would serve 6model better.
Today MoarVM is revealed. It's a VM for 6model.
Imagine taking all the lessons learned about Perl 6 and VMs, and factoring them into a project like this. jnthn's article goes on to list all the things that MoarVM does:
- lightweight and focused
- quick and easy build
- JIT compilation (to be explored in the future)
The really nice aspects of this can be read between the lines: MoarVM is lightweight, so it starts up faster than the JVM, and even before the real optimization and JIT work begins, its runtime performance already shows some promise.
There's no historical baggage, and there's not a sense that the project is fighting technical debt when moving forward. Since MoarVM is designed for nqp from day one — in a very real sense, MoarVM is an nqp backend — we're guaranteed both a nice direct focus on Rakudo, as well as the ability to support other higher-level languages long-term. Things are factored in a way that we can build Perl 6 on. Hard-earned VM lessons from 2001 up until today are coded into the foundation.
In other words, MoarVM reboots the whole VM idea for Perl, based on experience with Parrot, without many of the flaws of Parrot.
Of course, work is still very much ongoing on other backends as well. With nqp bootstrapped on the JVM, about to be bootstrapped on MoarVM, and likely running on JavaScript soon, we're looking at a very interesting 2013.
May 31, 2013 19:23
Over the course of the last year, we’ve been working to make both NQP and Rakudo more portable. This has primarily been done to enable the JVM porting work. While the JVM is an important platform to target, and initial work suggests it can give us a faster and more stable Rakudo, there are some use cases, or users, that the JVM will not cater to so well. Startup time will likely remain a little high for one-liners and quick scripts. Other potential users simply don’t want to use the JVM, for whatever reason. That’s fine: there’s more than one way to do it, and our strategy has been to enable that by adding support for the JVM without breaking support for Parrot. Additionally, pmurias will be working on a JavaScript backend for a GSoC project.
Today I’d like to introduce some work that, all being well, will lead to an additional “way to do it” arriving over the next several months. A small team, composed of myself, diakopter, japhb and masak, have been quietly working on taking the design of the 6model object system and building a runtime around it. Thus, we’ve created the “Metamodel On A Runtime” Virtual Machine, or the “MOAR VM”, which we’ve taken to writing as “MoarVM”.
This is not a release announcement. At present, MoarVM runs neither NQP nor Rakudo, though a cross-compiler exists that allows translating and passing much of the NQP test suite. We’re revealing it ahead of YAPC::NA, so it can be openly discussed by the Perl 6 team. The goal from the start has been to run NQP, then run Rakudo. The JVM porting work has established the set of steps that such an effort takes, namely:
- Build an NQP cross-compiler that targets the desired platform. Make it good enough to compile the MOP, built-ins and the classes at the heart of the regex/grammar engine. Make it pass most of the NQP test suite.
- Make the cross-compiler good enough to cross-compile NQP itself.
- Close the bootstrap loop, making NQP self host on the new platform.
- Do the Rakudo port.
At the moment, the JVM work is well into the final step. For MoarVM, we’ve reached the second step. That is to say, we already have a cross-compiler that compiles a substantial range of NQP language constructs into MoarVM bytecode, including the MOP, built-ins and regex-related classes. Around 51 of the NQP test files (out of a total of 62) pass. Work towards getting the rest of NQP to cross-compile is ongoing.
Since anybody who has read this far into the post probably has already got a whole bunch of questions, I’m going to write the rest of it in a question-and-answer style.
What are the main goals?
To create a VM backend for Rakudo Perl 6 that:
- Is lightweight and focused on doing exactly what Rakudo needs, without any prior technical or domain debt to pay off.
- Supports 6model and various other needs natively and, hopefully, efficiently.
- Is a quick and easy build, with few dependencies. I was rather impressed with how quick LuaJIT can be built, and took that as an inspiration.
- Enable the near-term exploration of JIT compilation in 6model (exploring this through invokedynamic on the JVM is already underway too).
What’s on the inside?
So far, MoarVM has:
- An implementation of 6model. In fact, the VM uses 6model as its core object system. Even strings, arrays and hashes are really 6model objects (which in reality means we have representations for arrays and hashes, which can be re-used by high-level types). This is the first time 6model has been built up from scratch without re-using existing VM data structures.
- Enough in place to support a sizable subset of the
nqp:: op space. The tests from the NQP test suite that can be translated by the cross-compiler cover a relatively diverse range of features: the boring easy stuff (variables, conditionals, loops, subs), OO stuff (classes, methods, roles, mixins, and, naturally, meta-programming), multiple dispatch, most of grammars (including LTM), and various other bits.
- Unicode strings, designed with future NFG support in mind. The VM includes the Unicode Character Database, meaning that character name and property lookups, case changing and so forth can be supported without any external dependencies. Encoding of strings takes place only at the point of I/O or when a Buf rather than a Str is requested; the rest of the time, strings are opaque (we’re working towards NFG and also ropes).
- Precise, generational GC. The nursery is managed through semi-space copying, with objects that are seen a second time in the nursery being promoted to a second generation, which is broken up into sized heaps. Allocations in the nursery are thus “bump the pointer”, the copying dealing with the resulting fragmentation.
- Bytecode assembly done from an AST, not from a textual format. MoarVM has no textual assembly language or intermediate language. Of course, there’s a way to dump bytecode to something human-readable for debugging, but nothing to go in the opposite direction. This saves us from producing text, only to parse it to produce bytecode.
- IO and other platform stuff provided by the Apache Portable Runtime, big integer support provided by libtommath, and re-use of existing atomic ops and hash implementations. We will likely replace the APR with libuv in the future. The general principle is to re-use things that we’re unlikely to be able to recreate ourselves to the same level of quality or on an acceptable time scale, enabling us to focus on the core domain.
What does this mean for the Rakudo on JVM work?
Relatively little. Being on the JVM is an important goal in its own right. The JVM has a great number of things in its favor: it’s a mature, well-optimized, widely deployed technology, and in some organizations the platform of choice (“use what you like, so long as it runs on the JVM”). No matter how well Moar turns out, the JVM still has a couple of decades head start.
Additionally, a working NQP on JVM implementation and a fledgling Rakudo on JVM already exist. Work on making Rakudo run, then run fast, on the JVM will continue at the same kind of pace. After all, it’s already been taking place concurrently with building MoarVM. :-)
What does this mean for Rakudo on Parrot?
In the short term, until MoarVM runs Rakudo well, this shouldn’t really impact Rakudo on Parrot. Beyond that is a more interesting question. The JVM is widely deployed and battle-hardened, and so is interesting in its own right whatever else Rakudo runs on. That’s still not the case for Parrot. Provided MoarVM gets to a point where it runs Rakudo more efficiently and is at least as stable and feature complete, it’s fairly likely to end up as a more popular choice of backend. There are no plans to break support for Rakudo on Parrot.
Why do the initial work in private?
There were a bunch of reasons for doing so. First and foremost, because it was hard to gauge how long it would take to get anywhere interesting, if indeed it did. As such, it didn’t seem constructive to raise expectations or discourage work on other paths that may have led somewhere better, sooner. Secondly, this had to be done on a fairly limited time budget. I certainly didn’t have time for every bit of the design to be bikeshedded and rehashed 10 times, which is most certainly a risk when design is done in a very public way. Good designs often come from a single designer. For better or worse, MoarVM gets me.
Why not keep it private until point X?
The most immediate reason for making this work public now is because a large number of Perl 6 core developers will be at YAPC::NA, and I want us to be able to openly discuss MoarVM as part of the larger discussions and planning with regard to Perl 6, NQP and Rakudo. It’s not in any sense “ready” for use in the real world yet. The benefits of the work being publicly known just hit the point of outweighing the costs.
What’s the rough timeline?
My current aim is to have the MoarVM backend supported in NQP by the July or August release of NQP, with Rakudo support to come in the Rakudo compiler release in August or September. A “Star” distribution release, with modules and tools, would come after that. For now, the NQP cross-compiler lives in the MoarVM repository.
After we get Rakudo running and stabilized on MoarVM, the focus will move towards 6model-aware JIT compilation, improving the stability of the threads implementation (the parallel GC exists, but needs some love still), asynchronous IO and full NFG string/rope support.
We’ll have a bunch of the right people in the same room at YAPC::NA, so we’ll work on trying to get a more concrete roadmap together there.
Where is…
May 31, 2013 19:15
On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the May 2013 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball and Windows .MSI for the May 2013 release are available from http://rakudo.org/downloads/star/ .
In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2013.05 of the Rakudo Perl 6 compiler, version 5.3.0 of the Parrot Virtual Machine, plus various modules, documentation, and other resources collected from the Perl 6 community.
Some of the new features added to this release include:
- The ?-quantifier on regex captures now returns a single Match object (formerly returned an array). Use
** 0..1 to get the old behavior.
- Failed matches return Nil instead of a false Match object.
- Rakudo warns when pure expressions are used in sink context
- .substr(…) now correctly accepts whatever-star closures
- Implemented shellwords postcircumfix (%h<< $x ‘foo bar’ >>)
- Defining operators spelled like the empty string is now illegal
- Array interpolations now properly do LTM
- Autothread “none” and “all” junctions before “any” and “one”
- Helpful error if you write “else if”/”elif” instead of “elsif”
- Throw exception if a Range is used as a Range endpoint
- Corrected argument order in IO.seek
- wrap low level VM objects in ForeignCode, allowing perl6 OO calls on them
- for loops are eager again
- add link and symlink to IO
- add Capture.Bool()
- improvements to DUMP()
- various optimizations in the optimizer and the runtime
- smartmatch against list now supports Whatever wildcards
- IO::Spec, a port of Perl 5′s File::Spec
- regex special characters can be used as delimiters
- allow slice with :exists adverb on hashes
- added 125 extra opening/closing bracket-pairs
This release also contains a range of bug fixes, improvements to error reporting and better failure modes.
The following features have been deprecated or modified from previous releases due to changes in the Perl 6 specification, and are planned to be removed or changed as follows:
postcircumfix:<[ ]> and postcircumfix:<{ }> will become multi-subs rather than multi-methods. Both at_pos and at_key will remain methods.- Unary hyper ops currently descend into nested arrays and hashes. This will change to make them equivalent to a one-level map.
- The Str.ucfirst builtin is deprecated; it will be replaced by Str.tc.
- Leading whitespace in rules and under :sigspace will no longer be converted to
<.ws>. For existing regexes that expect this conversion, add a <?> in front of leading whitespace to make it meta again.
- The result of failed matches will be Nil instead of a Match object returning boolean False.
There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Some of the not-quite-there features include:
- advanced macros
- threads and concurrency
- Unicode strings at levels other than codepoints
- interactive readline that understands Unicode
- non-blocking I/O
- much of Synopsis 9
There is an online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo and other Perl 6 implementations.
In many places we’ve tried to make Rakudo smart enough to inform the programmer that a given feature isn’t implemented, but there are many that we’ve missed. Bug reports about missing and broken features are welcomed at rakudobug@perl.org.
See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. A draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf in the release tarball.
The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.
May 31, 2013 04:04
May 30, 2013
Before I begin, I apologize for being really late with this article. I had this article ready for really long time, but for some reason forgot to commit it.
I would like to thank entire Perl 6 community for providing very good product, and even providing already compiled builds I can test stuff on Perl 6 Community Development Server.
Rakudo Perl 2013.05 was released. This not Rakudo Star, and according to the policy, it isn’t mentioned on Rakudo webpage, to avoid confusion with Rakudo Star. Every change mentioned here is in.
Incompatible changes
Perl 6 specification
New features
Perl 6 specification
STD.pm6
Rakudo Perl
-
Slice adverbs support false booleans.
You can write code like %a<b> :!exists to check whatever the element doesn’t exist.
You can also write code like %a<b> :!delete which does nothing, but can be used for conditionally deleting hash keys (for example %a<b> :delete(%a<c> :!exists)).
-
dir works properly on pathes with volumes.
This affected platforms such as Windows (actually, is there any platform other than Windows that Rakudo supports and has volumes).
-
.hash accepts optional :type and :of named parameters.
-
spurt is not allowed in restricted setting.
Writing to file in restricted setting doesn’t make any sense, so it’s disallowed now. Please note that restricted setting was never meant as sandbox (you can easily ignore it by calling nqp:: or pir:: methods directly), but rather for services that execute code, to provide better error reporting for features that are disallowed anyway on higher level.
-
link, symlink, and chmod not allowed in restricted setting.
Ditto, but with different commit.
-
Pod::Block::Declaractor.gist now works.
Previously, gist method had invalid signature, and failed.
-
List and Parcels have .Set and .Bag method.](https://github.com/rakudo/rakudo/commit/a134b18ca72eaf5c902250674b6b8aa475748fe7)
It works just like set and bag subroutine, except as a method.
-
:exists works with slices.
It’s allowed by a spec. Previously, it shown error “Cannot use exists adverb with a slice”.
-
slurp supports IO::Handles.
-
Implemented IO::Handle.slurp.
It’s a logical step, considering slurp as a function already works.
-
Perl 5 variable $, only errors when it’s obvious.
-
$% and $@ itemizers are now allowed.
You don’t have to type $(% and $(@ for something that actually makes sense, just because Perl 5 didn’t allowed it. In Perl 5, these work similarly to \% and \@.
May 30, 2013 07:00
May 29, 2013
Howdy,
We have accepted 3 GSoC student proposals!
Javascript backend for Rakudo Perl 6 [0] by Paweł Murias (Poland)
Updating parrot-libgit2 to latest libgit2 release [1] by Saurabh Kumar (India)
Improve App::Parrot::Create [2] by Denis Boyun (Ukraine)
The community bonding period starts now. Please welcome our students
and give them constructive criticism about their proposals. They are
extremely intelligent students, but they do not have the years of real
world experience that many potential mentors posses.
If you are interested in helping mentor any of these projects (which
could be anything from answering a question on IRC occasionally to
being a dedicated mentor meeting with the student on a daily or weekly
basis), please become a mentor via Melange [3].
The above projects could use help from people that know about Perl 5,
Perl 6, Parrot, libgit2, C, the unix command-line, nasal demons and
the various portability Elder Gods that must be appeased to Make
Things Work.
Students can always use more encouragement, so please help me welcome
them to our awesome community and make sure they are constantly
learning and making progress.
If you ever want to know "what happens next", I highly recommend
joining the #gsoc IRC channel on Freenode, where you can type !next
and various other commands that will be answered by a properly trained
bot.
More details soon!
Duke
PS: We have at least two more proposals that I feel are worth funding
directly as grants from Parrot Foundation and/or The Perl Foundation.
I encourage any students who were not accepted to submit their
proposals directly to the previously mentioned organizations so it can
be decided if they should be funded directly.
[0] http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/pmurias/9002
[1] http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/saurabh_kgp/11002
[2] http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/chob_rock/9001
[3] http://www.google-melange.com/gsoc/homepage/google/gsoc2013
--
Jonathan "Duke" Leto <jonathan@leto.net>
Leto Labs LLC http://letolabs.com
209.691.DUKE http://duke.leto.net
@dukeleto
May 29, 2013 00:22
May 22, 2013
Jimi Hendrix, deceased, drugs.
Janis Joplin, deceased, alcohol.
Mama Cass, deceased, ham sandwich.
-- Austin Powers (making a list of friends from the Summer of Love)
On behalf of the Parrot team, I'm proud to announce Parrot 5.4.0, also known
as "Austin Parrot". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages, and currently focusing on Perl 6.
Parrot 5.4.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/5.4.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.
Parrot 5.4.0 News:
- Core
+ Implemented the coth() and acot() math functions.
+ Fixed chomp to only trim a newline when it ends the string. [GH #958]
+ Added readlink() and Parrot_file_readlink(), with tests. [GH #967]
- Build
+ Parrot now detects the CPU model on Linux systems, as well as
detecting more CPU models on BSD, Cygwin, Solaris, Win32, and Darwin.
ARM v7 is also now recognized. [GH #962]
- Documentation
+ Threads examples now have proper POD sections and useful descriptions
with links to references.
+ Added main description for Task PMC.
+ Added descriptions to trig methods in Float PMC.
- Tests
+ Added improved test coverage targets "cover_new" and "fullcover_new".
+ Improved tests for acot(), coth(), acot() math functions.
+ Added tests for options passed to debugger.
+ Updated native PBC test files for string, number, and integer,
which resolved 11 TODOs in the test suite. [GH #959]
+ Fixed test for the auto/arch config step.
- Release process
+ Added message digests to crow.pir.
+ Added in release.json: "release.type" can be "devel" or "supported".
+ Refactored common code to sub in auto_release.pl.
- Community
+ Parrot is part of the Hackathon at YAPC::NA::2013, in Austin, TX, USA!
http://www.yapcna.org/yn2013/wiki?node=Hackathons
+ Parrot has been accepted to Google Summer of Code 2013!
+ Currently there are two high-quality proposals being worked on:
https://gist.github.com/sa1/5468408 - parrot-libgit2
https://gist.github.com/denisboyun/5472762 - App::Parrot::Create
The SHA256 message digests for the downloadable tarballs are:
4e37686911b446f5e5f2c0aa62138988ba0c411d2c5e2ba231d1a3421a85ad10 parrot-5.4.0.tar.gz
91d0e46fe3ef08e692e80756f26ee0e7311fe58e49d6c31f3f5180d4eb475696 parrot-5.4.0.tar.bz2
Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 18 Jun 2013.
Enjoy!
May 22, 2013 18:51
May 18, 2013
This week, the most notable change would be Rakudo sort of working on Java Virtual Machine. It still needs lots of improvements, but it can already run “Hello, world” program, what is important milestone in Rakudo for JVM.
Incompatible changes
Perl 6 specification
STD.pm6
New features
Perl 6 specification
STD.pm6
-
set<a b c> has special error message.
This syntax wasn’t useful for anything, except for perhaps obfuscation. It meant creating empty set, and getting keys 'a', 'b', and 'c' from it. As the empty set doesn’t have these keys, it was returning False 3 times.
As the behavior doesn’t make any sense, you probably wanted to use set <a b c> instead. The old behavior is still accessible using set()<a b c>, but I don’t think you would want to use this.
-
Perl 5 grammar allows version in require statement.
In Perl 5, you can write code like require 5.010 do make runtime version check (as opposed to use 5.010 which is compile time check). The Perl 5 grammar, included as part of STD.pm6 now supports this syntax.
-
FINAL and COMPOSE phases is now recognized.
Now that these phases are in specification, the grammar was modified to recognize them.
Rakudo Perl
-
JVM version of Rakudo supports simplest “Hello, world!” program.
-
Fixed infinite recursion in IO::Spec::Unix.rel2abs.
rel2abs made infinite recursion if neither $path or $base were absolute.
-
infix:<~> is constant folded.
-
<prefix:<~> is constant folded.
-
String comparison operators are constant folded.
-
String bitwise operators are constant folded.
-
defined is constant folded.
-
Set, KeySet, Bag, and KeyBag have default method.
When reading undefined property of those classes, default method contains value that you should receive. In case of sets, it’s False. For bags, it’s 0.
-
Regexpes support more delimiters.
Previously, Rakudo didn’t accept regexpes like m^42^ or m€cash€. Now, they are allowed, just like in Perl 5.
Niecza
May 18, 2013 07:00
May 14, 2013
It’s time for another progress update on the ongoing JVM work. Last time I posted here, we’d reached the point of having a self-hosting NQP ready to merge into the master branch of the NQP repository. That has now happened, so the May release of NQP will come with support for running on the JVM (note, this does not mean the May release of Rakudo will come with this level of capability, and a JVM-based Star release with modules is further still!) . In this post, I will discuss some of the things that have happened in the last few weeks and also try to answer some of the questions left behind in the comments last time.
The Rakudo Port
With NQP pretty well ported (there are some loose ends to tie up, but it’s pretty capable), the currently ongoing step is porting Rakudo. At a high level, Rakudo breaks down into:
- The core of the compiler itself: the grammar (for parsing), actions (which assign semantics to the things we parsed), world (which takes care of the declarative aspects of programs), optimizer (tries to cheat without getting caught) and a few other small pieces to support all of this. This is written in NQP.
- The Perl 6 MOP (meta-object protocol) implementation, which defines what classes, roles, enums, subsets and so forth mean. This is also written in NQP.
- The bootstrap, which uses the MOP to piece together various of the core Perl 6 types. It does Just Enough to let us start writing Perl 6 code to define the rest of the built-ins. Also written in NQP.
- The core setting, which is where the built-in types, operators and methods live. This is written in Perl 6.
- A chunk of per-VM code that does lower-level or performance-sensitive things.
The first 3 of these need…an NQP compiler. And, it turns out that we have one of those hosted on the JVM these days. But could it handle compiling the Perl 6 grammar? It turns out that, after some portability improvements to the various bits of Perl 6 compiler code, the answer was a resounding “yes”. In fact, the grammar was compiled down to JVM bytecode sufficiently accurately that I didn’t encounter a single parse failure on the entire CORE.setting (though there were lots of other kinds of failures that took a lot of work – more on that in a moment). As for the rest of the compiler code, there were bits of lingering Parrot-specific stuff throughout it, but with a little work they were abstracted away. By far the hardest was BOOTSTRAP, which actually runs a huge BEGIN block to do its setup work and then pokes the results into the EXPORT package. This is kinda neat, as it means the setup work is done once when building Rakudo and then serialized. Anyway, onto the next pieces.
Compiling the Perl 6 setting depends on the Perl 6 compiler working. Since the first thing the setting does is use the bootstrap, which in turn uses the MOP, it immediately brings all of the above three pieces together. While we talk about “compiling” the setting, there’s a little more to it than that. Thanks to various BEGIN time constructs – such as traits, constant declarations and, of course, BEGIN blocks – all of which show up in the CORE setting – we actually need to run chunks of the code during the compilation process. That’s right – we run bits of the file we’re in the middle of compiling while we’re compiling it. Of course, this will be nothing new to Perl folks – it’s just most Perl programmers probably don’t worry about how on earth you implement this. :-) Thankfully, it’s a solved problem in the NQP compiler toolchain, and the stuff that makes NQP BEGIN blocks work largely handles the Rakudo ones too.
Anyway, all of this means that even getting the setting to finish the parse/AST phase requires doing enough to run the various traits and so forth. And that in turn brings in the fifth piece: the per-VM runtime support. This includes signature binding, container handling and a few other bits. Thankfully, it no longer involves multiple dispatch, since that is written in NQP these days (apart from some caching infrastructure, which is shared with NQP’s multiple dispatch, and thus was already done). Getting through the parse/AST phase of the setting doesn’t need all of the runtime support to be implemented, but it did require a decent chunk of it. Of course, at the start everything is missing, so getting from line 1 to line 100 was O(week), from 100 to 1000 O(day) and each thousand or so from there O(hour). It’s 13,000 or so lines in all.
The parse/AST step is just the first (though biggest) phase of compiling Perl 6 code, however. Next comes the optimizer, followed by code generation. In theory, the optimizer could have been bypassed. I planned to do that, then discovered it basically worked for the set of optimizations that didn’t need signature binding to participate in the analysis, so I left it in. Code generation is part of the backend, and so is shared with NQP. So it shoulda just worked, right? Well, yes, apart from code generation is the place where nqp:: ops get resolved to lower level stuff. And Perl 6 uses a lot more of them than NQP. Note that they only have to be mapped to somewhere; the JVM is late bound enough that it won’t complain unless you actually hit the code path that tries to use something that is not yet implemented. In reality I did a bit of both: implementing those that would surely be hit soon or that were trivial, and leaving some others for later.
So, some time yesterday, I finally got to the point of having a CORE.setting.class. Yup, a JVM bytecode file containing the compiled output of near enough the entire Perl 6 core setting. So, are we done yet? Oh, no way…
Today’s task was trying to get the CORE setting to load. How hard could that be? Well, it turns out that it does a few tasks at startup, most of which hit upon something that wasn’t in place yet. Typically, it was more of the runtime support, though in some cases it was unimplemented nqp:: ops. Of course, there were a handful of bugs to hunt down in various places to.
Anyway, finally, this evening, I managed to get the CORE setting to load, at which point, at long last, I could say:
perl6 -e "say 'hello world, from the jvm'"
hello world, from the jvm
Don’t get too excited just yet. It turns out that many other simple programs will happily explode, due to hitting something missing in the runtime support. There’s still plenty of work to go yet (to give you an idea, trying to say a number explodes, and a for loop hits something not yet implemented), but this is an important milestone.
Interop
A couple of the comments in response to my last post asked about interop with Java. There’s two directions to consider here: using Java libraries from Perl 6, and using Perl 6 code from Java. Both should be possible, with some marshalling cost, which we’ll no doubt need to spend some time figuring out how to get cheap enough it’s not a problem. It may well be that invokedynamic is a big help here.
The Java stuff from Perl 6 direction can probably be made fairly convenient to use by virtue of the fact that Perl 6 has a nice, extensible MOP. The fact the object you’re making a call on lives in Java land can be just a detail; we can hide it behind the typical method call syntax, and should even be able to populate a 6model method cache with delegation methods that do the argument mapping. I’m sure there will be plenty of interesting options there. I suspect we’ll want to factor it a little like NativeCall – some lower level stuff in the runtime, and some higher level sugar.
Going the other way will be more “fun”. I mean, on the one hand the marshalling is just “in the other direction”, which we’d need to do for values coming back from Java land anyway. But trying to work out how to make it feel nice from Java land could be trickier. I don’t believe the “.” operator is very programmable, which probably leaves us with string lookups or code-generated proxy thingies. Or maybe somebody will come up with a Really Great Solution that I hadn’t thought of.
My JVM related Perl 6 dev focus for now will be getting Rakudo to work decently and start getting Perl 6 modules working on the JVM also, but interop with Java land is certainly on the roadmap of things I think should happen. As with all things, I’m delighted to be beaten to it, but will work on it if it goes undone for too long. :-)
Performance?
The first thing to say on this is that it’s too early to have a really good idea. The final pieces of the gather/take transform (which has global consequences) have yet to land, which will certainly have some negative impact and will need to happen soon. At the same time, I’ve been very much focused on making things work on the JVM at all over making them especially clever or optimal. Numerous things can be done in ways that will not only perform better in a naive sense, but that will also be much easier for the JVM’s JIT to do clever stuff with. There are many, many things we will be able to do in this area.
Since I only have a sort-of-working Perl 6 compiler, I can’t say that much about Perl 6 performance. The only result I have to share is that the CORE setting parse completes in around a third of time that it does on Parrot (noting it’s not only about parsing, but also some code generation and running of stuff). This is not especially great – of course, we need to do better than that – but it’s certainly nice that the starting point before I really dig into the performance work is already a good bit faster.
The other result I have is NQP related. nwc10++ has been doing performance testing with a Levenstein benchmark of each commit to the NQP JVM work, which is really great in so far as it gives me a rough idea if I accidentally regress (or improve ;-)) something performance wise. There, we saw a larger factor performance win, around 15 times faster than the same program running in NQP on Parrot.
The big negative news on performance is startup time. Part of this is just the nature of the JVM, but I know another enormous part of it is inefficiencies that I can do something about. I’ve plenty of ideas – but again, let’s make things work first.
From Here
Things will be a little quieter from me over the next week and a bit, due to a busy teaching schedule. But now we have a fledgling Rakudo on JVM, and from here it’s going to be making it gradually more capable, first passing the sanity tests, then moving on to the spectests and the ecosystem. There are ways to help for the adventurous. Some ideas:
- Profile the code generation phase, which is one of the pieces that is slower than expected. Try to figure out why.
- Have a look at how multiple dispatch stuff is currently set up, and see if the dispatch logic could possibly be shuffled off behind invokedynamic.
- Try something. See how it explodes. See if you can fix it. (Yes, generic I know. :))
- Have a look at a “make install” target for Rakudo on JVM.
I’ll be speaking on the JVM work at the Polish Perl Workshop the weekend after next, and hope to have something a bit more interesting than “hello world” to show off by then.
May 14, 2013 23:29
May 11, 2013
So, well, what’s new. I’m going to answer - lots of changes and fixes. Even the Niecza, where lately development had slowed down got lots of fixes.
Incompatible changes
Rakudo Perl
-
Array references aren’t treated as arrays in regular expressions.
Following code used to find 1, 2, or 3.
my $array = [1, 2, 3];
my $number = prompt 'Give me a number!';
say "It's in array" if $number ~~ / ^ $array $ /;
It doesn’t do that anymore, because variables are expected to be stringified inside regexpes. If you want the old behavior, be explicit and write @$array instead of $array.
Please note that currently regular expressions are exception to this rule. This is going to be fixed in the future. Lots of code depends on regexpes to be not stringified, and first the warnings will have to be implemented. You are expected to use <$regexp> instead of $regexp.
Niecza Perl
New features
Rakudo Perl
-
IO::Spec::Win32 no longer warns when using relative pathes.
The check for initial prefix was using optional alternative (([a | b | c])?). Because of that, it was Nil if it matched nothing. Since the result of match was stringified just after it was matched, it returned a warning when ? matches nothing. It was fixed to do ([a | b | c]?) instead.
-
Added tests for IO::Path and IO::Spec to list of tests that should pass.
As IO::Path and IO::Spec were implemented, their tests are now ran when you request full test from roast test suite.
-
Interpolation of quotes in regexpes isn’t done twice anymore.
Previously, quotes inside RegExpes were doing interpolation two times.
-
match method is fast again.
https://github.com/rakudo/rakudo/commit/c255f1df the older commits) made it 10 times slower. The performance regression was fixed, and now match is as fast as it used to be.
-
wordcase method now accepts where which is Mu.
Previously, only Any values were accepted. While using Mu doesn’t make much sense, there is no reason to disallow it, so it’s allowed now.
-
Configuration instructions don’t say to use –gen-nqp.
It was pointless, since --gen-parrot automatically implied --gen-nqp. If you want Parrot, it’s safe to assume you want NQP for that Parrot too, right?
-
our hashes and arrays work.
If you had used our @array, and our %hash, the object wasn’t initialized properly, and it wasn’t working like array or hash. This made our work only with $scalars previously.
Previously, trying to use our with arrays or hashes caused strange uninitialized variable to appear. It didn’t have STORE method, so you couldn’t have put anything in this variable, and had Any instead of empty array of hash.
-
It’s now possible to augment empty list.
MONKEY_TYPING couldn’t be used to modify lists. Now it can.
Niecza Perl
May 11, 2013 07:00
May 05, 2013
On May 1st I learned what happens if you tell a bunch of people on the
Internet, not all of whom you've met before, that you're going to teach Perl (5
and 6) for free on an IRC channel for four hours.
It worked well. Actually, it worked well beyond my expectations.
Successes
Prepare. I had written a bunch of material; short-ish texts mixed with
exercises. I didn't hear any feedback about the texts, but people threw
themselves on the exercises. Which was lucky, because I basically didn't
have time to interact much with people. Everyone did their own thing.
Map. The week before the workshop, I was taking a walk, and it occurred
to me that the topics for the workshop should be laid out on a subway map.
That turned out to work unexpectedly well. People followed the structure,
I think. The material was pretty clear that they were free to skip/jump
around if they wanted, so some did that.
Github. People suggested fixes and improvements to the material during
the workshop. I handled the easy ones, and delegated what could be delegated.
If anything about the workshop felt 2013-futuristic, it was the fact that
participants were hacking on the workshop in real time, as it was being
played out. Github was totally right for this.
- tadzik++ did an exercise and got a solution that I didn't expect. We
agreed that a test was missing to force people to do the right thing,
so he went ahead and added it.
- LlamaRider++ added hyperlinks between all the files.
- grtodd++ made the subway map clickable.
- choroba++ added hyperlinks to a file that I added later.
Things learned
Newbies. This was never a workshop for total beginners. Still, we got
a number of those. I'm not sure how many. If there's a next time, I'll
want to add a track for the people who haven't done Perl before.
Exercises. Based on the tasks that people actually
solved, I got a
lot of feedback on what worked and what didn't. Which exercises served the
overall message of the course, and which ones didn't.
Message. After the workshop, I know much better what I wanted the
whole thing to be about. I think I can go back and make that even more
clear. I'm still surprised at how well it worked already the first time
around... but some bits in there can certainly be improved.
Message
So... what was the real message of the workshop? What is "masakism"?
Two things:
- Testing gets you far.
- My pain threshold for writing things without tests is half a screen of
code. If I have more than that, there's probably something I should
be writing tests for.
- That said, there's no need to be a fundamentalist about anything. I'm
just saying that tests are good, and still somewhat undervalued
sometimes. But it's always a question on spending time testing the
stuff that matters. Of course.
- Even with tests, there's no substitute for knowing what you're doing
and where you're going. Tests help a lot, but they don't write the
program for you.
- Keep it small and simple.
- "Inside every big interesting problem is a small interesting problem,
struggling to get out."
- The realization that you can write objects that focus entirely on
your core problem, is a strong one.
- People tend to bring in persistence too early, "polluting" their core
classes with database code. Hindering testability. This is backwards.
- Your program should be like an onion, with the most precious stuff
in the middle, and layers around that. Outer layers should point to
inner layers, but not vice versa.
Once more?
So, should we do another #masakism workshop?
Yes, maybe we should. People seemed to like this first one. I'm open to finding
a datetime for another one.
If you have any suggestions, get in touch.
May 05, 2013 21:22
May 04, 2013
Perl 6 gets more and more stable, less features are added, and the features that are left are worked on. Also, Rakudo Perl is being ported to JVM.
New features
Perl 6 specification
STD.pm6
Rakudo Perl
-
Implemented IO::Spec.
This class is intended to do OS-specific manipulation of pathes. It can get the localizations of special folders (for example home directory, root directory, temponary directory), and can manipulate pathes (by joining them, spliting them, converting them to absolute or relative, and so on).
-
Implemented IO::Path::<os subclasses>.
Thanks to this change, the pathes objects are OS specific. You can have path marked to be Windows path, so you can parse it in OS specific way, without having to use that OS.
-
Repeated shift is faster.
On large arrays, shift is now noticeably faster. For example, if array @a has 50_000 elements, running following snippet of code is processed within 2.5 seconds (and not 9.0 seconds).
@a.shift while @a;
-
Failed regex matches now return Nil.
Previously, failed regex matches were returning the Match object that stringified to #<failed match> and was converted to False.
-
&diag function in Test.pm accepts Mu.
This, I think is self explanatory. &diag is a debugging function, outputing its argument. Forbidding Mu didn’t made sense, so it’s now allowed.
-
Exporting of subroutines now works properly.
use Module qw(sub1 sub2) should now work.
-
Fixed dir() to work on Windows.
This makes panda work on Windows.
May 04, 2013 07:00
April 28, 2013
I have a problem:
Should I keep the 3 Perl Maven sites (
Perl 5 Maven,
Perl 6 Maven, and
Perl Maven) separate,
or should I merge them together?
Or maybe I should merge P5M into PM and leave P6M separate.
For the full article visit To merge or not to merge?
April 28, 2013 18:04
April 27, 2013
Continuing, still with commit IDs.
Incompatible changes
Perl 6 specification
DateTime doesn’t accept Callable timezones anymore. 5692ced333
Rakudo Perl
New features
Rakudo Perl
April 27, 2013 07:00
April 25, 2013
sheep hide at bad rains
bitterly modest slumber
its star to their rains
-- generated by one of the t2 solutions
How time flies. It's two months since I reviewed the first batch of
p6cc solutions. I solemnly swear the
next blog post will appear sooner than that.
## Generate rectangle haikus
Write a program that generates haikus. A haiku has three lines, where the first
and last lines have five syllables each, and the middle line has seven. For the
purpose of this exercise, each line consists only of letters and spaces.
The haikus generated have the additional requirement that each line be of equal
length. The length requirement should not be gamed in any way, for example by
padding lines with spaces.
since this one does it
rectangle plus a haiku
it serves as fine text
The program should attempt to generate a new haiku each time. The haiku should
consist of English words. If the words make sense in some kind of sentence
structure, that's considered a big bonus. Humor and/or deeper meanings are even
bonuser.
If you attempt to cheat at this task, you will be defeated by people who don't.
You're free to supply your own wordlist. How you count syllables is up to you,
and part of the task. You won't be challenged on trivial differences in
syllable counts.
It's OK for the program to run for a while, but it should preferably terminate
within a reasonable span of time.
I warmly recommend people to check out the
solutions to this one. People are all over the place, coming up with ways to generate a rectangular haiku.
Some patterns recur, though. Let's break things down into subtasks and discuss
them separately.
Word lists. Some contestants had their wordlists inlined in the code.
Others had it in a separate file. Some stored part-of-speech information
together with the word, or syllable count.
Sentence structure. Solutions are all across the spectrum, all the way from
"just spew words", via "generate something that could perhaps pass for a
sentence", all the way to "try really hard to be grammatical". In my view, being
more high-end on this bit pays off big in quality. But it also costs in code
complexity and program speed.
Search. This is a biggie, because it bleads through into the whole design of
the program. Some of the solutions simply brute-force it. Others generate stuff
randomly but store things in a hash table, guaranteeing that eventually some
triplet of lines will form a haiku of some length. Some algorithms do clever
things here, like turning problems around, going from "how many syllables does
this line have?" to "ok, give me a word with this many syllables". One solution
partitions the integers 5 and 7 in all possible ways, and hard-codes the
resulting table.
But the really interesting bit in all solutions is the syllable counting. This
is where the participants really differ in approach. One went with porting the
CPAN module for counting syllables. Someone else did the same, but put the
module on modules.perl6.org for everyone to use. Two contestants seem to have
come up with their own (flawed) syllable-counter. A few people evade the
syllable-counting by just storing the values along with the word lists. (Which
is fine.)
All in all, an interesting set of solutions to a fun problem. Somehow, after
reviewing all of these, I have the distinct feeling that a "best" solution
could be put together by combining the best bits of everyone's solutions. I
might just do that myself, if no-one beats me to it.
Next up: arranging wire crossings to rearrange wires!
April 25, 2013 20:50
April 24, 2013
So, as part of my $dayjob, I was teaching a Perl course. Then this happened:
* masak is teaching Perl 5 today! :)
<Heather> masak teach me
<masak> Heather: later; these people are paying me. :P
[...]
<masak> seriously though; if there were enough interest here on IRC, I
would totally do a 1-day Perl 5 IRC workshop, pro bono.
<masak> ditto a one-day Perl 6 IRC workshop.
<vaelxon> (Perl 6 IRC workshop)++
<masak> it'd be very awesome, I promise. I'd need some time to prepare.
but then we can grab a channel and basically party/learn for a
day together.
And that's now what's happening. It'll be a loose-knit group of us, joining up
in the #masakism IRC channel for four hours on May 1.
The course will be two-pronged; Perl 5 or Perl 6, whatever people prefer.
There will be prepared study material and exercises, but basically, people who
participate can go off on a tangent, just spend the four hours solving one of
the exercises really well, or solving a different exercise that they make up
themselves. People can focus on doing the exercises, or on discussing various
ways to solve them, or on discussing general principles of programming. Think
of the workshop as a hybrid of people writing/showing nice code, and people
discussing programming best practices.
Why am I doing this? Because as part of teaching Perl courses for $dayjob, I
realized how much I love teaching Perl. I literally spend those two course
days teaching Perl in a mild state of euphoria. I teach a lot of other courses,
and some of them are great to teach, but none of them are like Perl. I think
it's because I feel at home in the language, my brain has somehow "shaped
itself" around Perl, and the questions I get can often be answered with a
happy "let's find out!" where (a) we do some quick experiment in a terminal
window, and (b) I tend to know what we'll get before, so it's easy to explain.
The whipuptitude of Perl really works to one's advantage there. That, combined
with knowing my way around most of the perldoc pages and most of the special
variables. Oh, and I probably have a slight advantage being fairly deeply
involved in Perl 6 stuff, too.
Anyway, I absolutely love teaching Perl. Five and six. So much so that I'd
do it for free. So this is me doing it for free.
As I write this, there are 13 participants signed up for the course. This is
already twice as big as the typical Perl course I teach in a corporate setting.
It's also a wide range of people; from relative newbies to people who could
probably give the workshop better than me. I expect people further up on the
learning curve to step in and assist with people further down. But I hardly
need to say that; this is Perl, and we're a helpful bunch.
Here's more info about the workshop,
including when, where, what, and how to sign up.
April 24, 2013 22:37
April 20, 2013
It’s sixteen article in the series. The game goes so fast, and I still haven’t done pagination on my website (lazy as always). The scrollbar is still shorter (1 pixel on my PC) than on Planet Six, but most likely because of longer lines.
NQP was bootstrapped on JVM. That means it can compile itself. It still doesn’t support Perl 6 perfectly, because some code still uses pir::, but it progressed really far. Also, the new version of Rakudo Perl, 2013.04 “Albany” was released. Every improvement mentioned here, is in.
I’ve decided to experiment with commit links this time. This way, you can click (or press, or whatever, depends on your device, to get code of this change).
Potentially incompatible changes
Perl 6 specification
New features
Rakudo Perl
IO::Path is closer to actual spec. 24532aca13- Added
Bool method to Capture. 9f873d89e3
List.ACCEPTS now has special Whatever handling. f17d339185do...for doesn’t propose repeat...for now. 41a92e808e- Documented build memory requirements (1GB of memory). 4109be8905
require PATH|$variable|Module:file(PATH) syntaxes are now accepted. c946d0f7f7- Removed reporting variable name for uninitialized variable warnings because it was producing too many false positives. c39b887388
- Replaced more
pir:: to nqp::. In particular, getting env variables is now done using NQP, not Parrot directly. e9e1b5d962
April 20, 2013 07:00
April 18, 2013
The work to get NQP running and bootstrapped on the JVM has reached an interesting milestone, and I thought I’d take a few moments from working on it to talk about what has taken place since my last post here, as well as taking a look at what will be coming next.
Getting NQP on JVM Bootstrapped
When I last posted here, I had reached the point of having an NQP cross-compiler to the JVM that covered quite a lot of NQP’s language features. Cross-compilation meant using NQP on Parrot to parse NQP source, build an AST, and then turn the AST into a kind of JVM assembly language. This in turn was then transformed into a JVM bytecode file by a small tool written in Java, which could then be executed on the JVM.
Since the last post, the cross-compiler became capable of cross-compiling NQP itself. This meant taking the NQP source files and using the cross-compiler to produce an NQP that would run on the JVM – without depending on Parrot. This also enabled support for eval-like scenarios. I reached this stage last month; the work involved tracking down a range of bugs, implementing some missing features, and doing a little more work to improve NQP’s portability. So, we had an NQP that ran on the JVM. Port done? Well, not quite.
A little vacation later, I dug into the next stage: making NQP on the JVM able to compile (and thus reproduce) itself. While I’d already implemented the deserialization used to persist meta-objects (representing classes, grammars and so forth), for this next step I had to implement the serialization side of things. Thankfully, there are a bunch of tests for this, so this was largely “just” a matter of working through making them pass. Finally, it was time to work through the last few problems, and get NQP on JVM able to build the NQP sources – and therefore able to compile new versions of itself. I decided to do this work as part of moving the JVM support into the main NQP repository.
Since we’ve only had NQP running on one backend (Parrot) up until now, certain aspects of the repository structure were not ideal. Before starting to bring in the JVM support, I first did a little bit of reorganization to segregate the VM specific components from the VM independent ones. Happily, much of NQP’s implementation falls into the latter category. Next came gradually building up the bootstrapping build process, working a file at a time, tracking down any issues that came up. This was a little tedious, especially given a couple of the problems were separate compilation leakages (where things from the running compiler would get confused with the version of the compiler that it was compiling). It was pretty clear that this was the problem from the errors I was seeing, but such problems show up long after things actually go wrong, requiring some careful analysis to hunt down. With those leaks plugged, and a few other relatively small bugs fixed, I had a working NQP on JVM…compiled by NQP on JVM.
The work from there has been to fill out the rest of the build process, adding in the second bootstrap stage and the test targets. The good news: the NQP produced by NQP on JVM passes all the tests that the original cross-compiled version did, so we’ve got no regressions there as a result of the bootstrap.
This work is currently sat in the jvm-support branch of the NQP repository. After the upcoming NQP release, it will be merged.
Supporting invokedynamic
Amongst all of this progress, we’ve also gained infrastructure to support using the invokedynamic instruction. This is a mechanism that enables those implementing non-Java languages on the JVM to teach its JIT about how their dispatch works. Most of the hard work here was done by donaldh++. I’d initially built things using BCEL in order to do code generation. While it served well up to a point, it turns out that ASM has much better support for invokedynamic, as well as being a little faster. So, donaldh got things switched over, and I soon was able to emit invokedynamic.
So far, we are not using it a great deal (just for making QAST::WVal compile to something a bit – or potentially a lot – cheaper), but in the future it will be used for things you’d typically think of as invocations (sub and method calls). I’ll write in more detail about it as things evolve.
What next?
With NQP now ported, the focus will fall on Rakudo itself. Quite a lot of preparations have already been made; for example, many pir:: ops have been replaced with nqp:: ones, multiple dispatch has been ported to NQP from C (fixing some bugs along the way), the way HLL boundaries work has been updated to cope with a fully-6model world (this also let me fix a long-standing introspection bug).
The path through getting Rakudo ported will largely follow the build order. This means starting with the module loader, then the compiler, followed by the MOP and its bootstrapping. After that comes the setting – the place where the built-ins live. There’s around 13,650 lines worth of that, so of course I expect to take it a little at a time. :-) I’ll try to remember to get a progress update here in a couple of weeks time.
April 18, 2013 16:37
A recent discussion on
IRC prompted me to share a small but neat trick with you.
If there are things you want to do quite often in the Rakudo
REPL (the interactive "Read-Evaluate-Print Loop"), it makes sense to create a
shortcut for them. And creating shortcuts for often-used stuff is what
programming languages excel at, so you do it right in Perl module:
use v6;
module REPLHelper;
sub p(Mu \x) is export {
x.^mro.map: *.^name;
}
I have placed mine in $HOME/.perl6/repl.
And then you make sure it's loaded automatically:
$ alias p6repl="perl6 -I$HOME/.perl6/repl/ -MREPLHelper"
$ p6repl
> p Int
Int Cool Any Mu
>
Now you have a neat one-letter function which tells you the parents of an
object or a type, in method resolution order. And a way to add more shortcuts
when you need them.
April 18, 2013 04:54
April 17, 2013
We are stardust.
Billion year old carbon.
We are golden.
Caught in the devil's bargain
And we've got to get ourselves back to the garden.
(To some semblance of a garden.)
-- "Woodstock", by Joni Mitchell
On behalf of the Parrot team, I'm proud to announce Parrot 5.3.0, also known
as "W00tstock Parrot". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages, and currently focusing on Perl 6.
Parrot 5.3.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/5.3.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.
Parrot 5.3.0 News:
- Build
+ Files generated by `make cover` are now correctly cleaned by `make`
and ignored by `git`.
- Tests
+ Internal testing of the Configure probe for Fink now works correctly
with the --verbose flag.
+ Tests added for .sort method of ResizableFloatArray and
ResizableIntegerArray. [GH #926], [GH #927]
+ Benchmarks added for .sort methods of various Array objects.
[GH #175]
+ Coverage analysis added for pbc_disassemble.
The SHA256 message digests for the downloadable tarballs are:
79d6f1fe20645b0afbc496cd0d7850a78b8940230e7637c5356d780f5aa1750b parrot-5.3.0.tar.gz
4cff32521c79d8a783ad57d9a13e205ea3c1b1585085e0da80138b58b77d0ed5 parrot-5.3.0.tar.bz2
Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 21 May 2013.
Enjoy!
April 17, 2013 19:53
April 14, 2013
I apologize I forgot about Perl 6 changes article yesterday. This ought not to happen, but it did. As for reason why it happened, well, I guess I will tell you anyway - it’s that game.
New features
Perl 6 specification
Parcels are allowed to have 0 elements.
Rakudo Perl
.DUMP method can detect recursion for hashes.- Added
ForeignCode for non-Perl 6 code objects (like C code).
- Removed lots of
pir:: usage for JVM port of Rakudo.
- Semicolon after keyword that takes a block, like
PRE ; now has specific error message.
- Added
symlink and link.
require can take paths.
Niecza Perl
- Added special cases for
big_pow when either of values is -1, 0, or 1.
April 14, 2013 07:00
April 06, 2013
You may have noticed may page went down for some time. This appears to be connected to newest domain change. In order to be more… modern, I’ve moved the repository to glitchmr.github.io, but that made entire page to go down. I apologize for inconvience, but it didn’t affected Planet Six, so I think it’s fine.
As always, I’m going to describe changes in Perl 6.
New features
Perl 6 specification
- Renamed
weekday-of-month to day-of-week-in-month.
Rakudo Perl
- Fixed signature of
Pod::Block.gist, so it works properly with multis.
for loops are eager now.- Added
Buf.perl.
- Fixed
Buf.gist, so it works properly with multis.
April 06, 2013 07:00
April 01, 2013
After or while a compiler parses a program, the compiler usually translates
the source code into a tree format called Abstract Syntax Tree, or
AST for short.
The optimizer works on this program representation, and then the code
generation stage turns it into a format that the platform underneath it
can understand. Actually I wanted to write about the optimizer, but noticed
that understanding the AST is crucial to understanding the optimizer, so let's
talk about the AST first.
The Rakudo Perl 6 Compiler uses an AST
format called QAST. QAST nodes derive from the common superclass
QAST::Node, which sets up the basic structure of all QAST
classes. Each QAST node has a list of child nodes, possibly a hash map for
unstructured annotations, an attribute (confusingly) named node
for storing the lower-level parse tree (which is used to extract line numbers
and context), and a bit of extra infrastructure.
The most important node classes are the following:
- QAST::Stmts
- A list of statements. Each child of the node is considered a separate
statement.
- QAST::Op
- A single operation that usually maps to a primitive operation of the
underlying platform, like adding two integers, or calling a routine.
- QAST::IVal, QAST::NVal, QAST::SVal
- Those hold integer, float ("numeric") and string constants
respectively.
- QAST::WVal
- Holds a reference to a more complex object (for example a class) which
is serialized separately.
- QAST::Block
- A list of statements that introduces a separate lexical scope.
- QAST::Var
- A variable
- QAST::Want
- A node that can evaluate to different child nodes, depending on the
context it is compiled it.
To give you a bit of a feel of how those node types interact, I want to
give a few examples of Perl 6 examples, and what AST they could produce. (It
turns out that Perl 6 is quite a complex language under the hood, and usually
produces a more complicated AST than the obvious one; I'll ignore that for
now, in order to introduce you to the basics.)
Ops and Constants
The expression 23 + 42 could, in the simplest case, produce
this AST:
QAST::Op.new(
:op('add'),
QAST::IVal.new(:value(23)),
QAST::IVal.new(:value(42)),
);
Here an QAST::Op encodes a primitive operation, an addition of
two numbers. The :op argument specifies which operation to use.
The child nodes are two constants, both of type QAST::IVal, which
hold the operands of the low-level operation add.
Now the low-level add operation is not polymorphic, it always
adds two floating-point values, and the result is a floating-point value
again. Since the arguments are integers and not floating point values, they
are automatically converted to float first. That's not the desired semantics for Perl 6; actually the operator
+ is implemented as a subroutine of name
&infix:<+>, so the real generated code is closer to
QAST::Op.new(
:op('call'),
:name('&infix:<+>'), # name of the subroutine to call
QAST::IVal.new(:value(23)),
QAST::IVal.new(:value(42)),
);
Variables and Blocks
Using a variable is as simple as writing
QAST::Var.new(:name('name-of-the-variable')), but it must be declared
first. This is done with QAST::Var.new(:name('name-of-the-variable'),
:decl('var'), :scope('lexical')).
But there is a slight caveat: in Perl 6 a variable is always scoped to a
block. So while you can't ordinarily mention a variable prior to its
declaration, there are indirect ways to achieve that (lookup by name, and
eval(), to name just two).
So in Rakudo there is a convention to create QAST::Block nodes
with two QAST::Stmts children. The first holds all the
declarations, and the second all the actual code. That way all the declaration
always come before the rest of the code.
So my $x = 42; say $x compiles to roughly this:
QAST::Block.new(
QAST::Stmts.new(
QAST::Var.new(:name('$x'), :decl('var'), :scope('lexical')),
),
QAST::Stmts.new(
QAST::Op.new(
:op('p6store'),
QAST::Var.new(:name('$x')),
QAST::IVal.new(:value(42)),
),
QAST::Op.new(
:op('call'),
:name('&say'),
QAST::Var.new(:name('$x')),
),
),
);
Polymorphism and QAST::Want
Perl 6 distinguishes between native types and reference types. Native types
are closer to the machine, and their type name is always lower case in Perl
6.
Integer literals are polymorphic in that they can be either a native
int or a "boxed" reference type Int.
To model this in the AST, QAST::Want nodes can contain
multiple child nodes. The compile-time context decides which of those is
acutally used.
So the integer literal 42 actually produces not just a simple
QAST::IVal node but rather this:
QAST::Want.new(
QAST::WVal(Int.new(42)),
'Ii',
QAST::Ival(42),
)
(Note that Int.new(42) is just a nice notation to indicate a
boxed integer object; it doesn't quite work like this in the code that
translate Perl 6 source code into ASTs).
The first child of a QAST::Want node is the one used by
default, if no other alternative matches. The comes a list where the elements
with odd indexes are format specifications (here Ii for
integers) and the elements at even-side indexes are the AST to use in that
case.
An interesting format specification is 'v' for void context,
which is always chosen when the return value from the current expression isn't
used at all. In Perl 6 this is used to eagerly evaluate lazy lists that are
used in void context, and for several optimizations.
April 01, 2013 00:22
March 30, 2013
The month is coming to end, and with that, I’m going to make yet another list of changes.
New features (like anything is other than that)
Perl 6 specification
- Added many
IO::Path methods: .contents, .is-absolute, .is-relative, .absolute, and .relative.
.path is now NOP that returns itself. For its previous behavior, use .Str.- Changed
Numeric.conjugate to Numeric.conj.
Rakudo Perl
+(23..2300000000) is optimized.--help reports -I and -M options.
March 30, 2013 07:00
March 27, 2013
Exactly two months remain until Polish Perl Workshop this year. In case you didn’t know, I’m shamelessly reposting the status update from our ACT website.
We’ve got the first sponsor offers, and about 15 talk submissions so far. Awesome! However, as we are well aware that most of you will wait to submit talks until there’s less than 2 days until the deadline (or later :)) I’m hereby announcing: The talk submission deadline for Polish Perl Workshop 2013 is 14 of April. That’s about two weeks to finally make up your mind and submit something. Please do! There’ll be only one chance to talk at the First Polish Perl Workshop: there’ll be no other :)
Of course, if you’d prefer to organize more of a hands-on event, we have the entire sunday dedicated to hackathon, BoFs, classes etc. If you want to submit something more lenghty, that’s the right way to do it.
In other interesting news, we started looking at conference t-shirts (yes, they’re free for all attendees). We have something on our mind that never happened before on a Perl conference (or at least I’ve never seen or heard of something like this). We’ll keep you informed.
March 27, 2013 11:56
March 24, 2013
I am not dead yet I can dance and I can sing
I am not dead yet I can do the Highland Fling
I am not dead yet No need to go to bed
No need to call the doctor Cause I'm not yet dead
-- Spamalot
On behalf of the Parrot team, I'm proud to announce Parrot 5.2.0, also known
as "Stuffed Parrot". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.
Parrot 5.2.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/5.2.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.
Parrot 5.2.0 News:
- Core
+ IO now only syncs buffers for the IO types where syncing makes sense.
= PIO_VF_SYNC_IO flag added
- Build
+ installable_pdump now has the correct rpath (blib corrected to lib).
- Libraries
+ Tcl/Glob.pir has been removed. (PGE/Glob.pir remains intact)
- Ecosystem
+ All Parrot tarballs are now symlinked to the 'all' directory,
regardless of their true homes ('devel' or 'stable'), to better
allow for automated downloads.
ftp://ftp.parrot.org/pub/parrot/releases/all/
The SHA256 message digests for the downloadable tarballs are:
1245d11f2b2ea44e6465aff6da5a533324d69b6eb3ddf7d84e81385ea95150ad parrot-5.2.0.tar.gz
0c538d780f9c70c510e142a8a663c30474125c9fcf9fe25d2129e68fc7baec8d parrot-5.2.0.tar.bz2
Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 16 Apr 2013.
Enjoy!
--
Bruce Gray (Util of PerlMonks)
March 24, 2013 13:58
March 23, 2013
I really should optimize my blog layout, so empty intro wouldn’t be ugly. But I guess I’m too lazy to do it right now, so I have to explain stuff. Every week, I list the list of changes in Perl 6 specifications, and its implementations. That’s just everything, boring, I know. Still, it doesn’t look that bad on Planet Six, so I guess I will continue to make these articles :-).
This week, the Rakudo 2013.03 was released. Every change mentioned here was implemented in it.
New features
Perl 6 specification
- Added
X::Range::InvalidArg.
Rakudo Perl
- Removed lots of
pir:: usage.
- Multi dispatch is now implemented in NQP, not C.
- Released Rakudo 2013.03.
March 23, 2013 07:00
March 19, 2013
It can be interesting to do operations on existing data structures in Perl 6, but without Input and Output,
that would have limited usability in the real world.
Therefore reading from files is a critical operation.
March 19, 2013 20:52
March 16, 2013
I would put nothing here, but the layout of blog would make this look ugly. So, well, like usually I put something here, even if it is meaningless. Just look below to see list of changes.
New features
Rakudo Perl
-
Closing brace, followed by newline and infix operator was parsed incorrectly. For example, following code.
my %a;
{ }
%a<b> = 1;
Was parsed like:
my %a;
{ } % a()<b> = 1;
Instead of correct.
my %a;
{ };
%a<b> = 1;
-
Done more pir:: to nqp:: replacements in order to help JVM port of Rakudo Perl.
-
Case insensitive matches with strings now work.
my $nonsense = 'asdf';
say "OK" if "ADSF" ~~ m:i/$nonsense/;
-
Typing else if and elif after if block shows “Please use ‘elsif’” error message.
-
&?ROUTINE variable is available inside regex foo { ... }.
-
The named params are optimized in compile time when possible.
-
X::Method::NotFound is raised even when dealing with "" (empty) method names.
-
Fixed proto of infix:<ne> in order to require Mu objects.
-
Stash.at_key method now works.
-
Postcircumfix calls can be cached now.
-
LEAVE is now fired on next/last in MapIter.
-
Exception is thrown when Range endpoint is used to create a Range.
Niecza Perl
Empty.gist returns nothing.
March 16, 2013 07:00
March 15, 2013
After a break of several months, I rebuild Rakudo on the server, a few days ago,
and regenerated the Perl 6 Maven pages. This promptly broke the site. The fix was easy, and as expected the bug was mine.
Today I decided to try Rakudo on my Windows machine as well, but I went for the easy
route and installed the pre-built msi file.
March 15, 2013 20:52
March 09, 2013
During making this update, I had lots of problems with Internet. I have managed to load commits history for every repository I check, but I haven’t checked commits. Will update this after Internet access will be more stable.
New features
Perl 6 specification
Rat.perl returns decimal number when denominator is normalizable to a power of 10.- Added FALLBACK methods.
- Included Failure in smartmatch passthroughs.
- Loosened relationship between
Nil and ().
- Added
List.permutations.
rat64 type has unsigned denominator.
STD.pm6
Nil isn’t suggested for empty list in error messages anymore.
Rakudo Perl
- Fixed expanding of variables that don’t exist when converting string to RegExp. For example,
"a" ~~ / <$blah> / is fixed.
Rat.perl returns decimal number when denominator is normalizable to a power of 10.- Removed lots of
pir:: usage in order to help porting Rakudo to JVM.
Rat.perl provides angle brackets around fractional forms.- Added missing shellwords postcircumfixes (
<< >> and « »).
- Generalized “augment” check, and made error message more generic.
- User defined circumfixes are parsed just like STD.pm6 does.
- Added colorcircumfixes.
- Rakudo now warns about integers starting with
0.
- Mistakes like
if(1 < 2) { ... } (where mistake is that if is called like a function) give good errors.
Nil isn’t suggested for empty list in error messages anymore.
Niecza Perl
Rat.perl returns decimal number when denominator is normalizable to a power of 10.CommonEnum.gist returns (Type) instead of Type() for undefined.- Restored
infix:<cmp> sorting for different types.
- Failed matches return
Nil.
March 09, 2013 08:00
March 06, 2013
For a little bit now I’ve had the ABC module working on both Niecza and Rakudo. I got a new Linux box yesterday; this morning I was playing around with my ABC tools there, trying to see if I could get them to work. I installed them with panda, so I was using ABC with Niecza by default on my MBP and with Rakudo by default on my new machine. I tried it with a file of my own compositions. Apparently I hadn’t tried this in a while, because it had several issues. And the two systems handled them completely differently.
On Niecza, I got this message:
Unhandled exception: Did not match ABC grammar: last tune understood:
A Hat for the Whales
d|:e~A3 BGEF|G2BG DG (3ABd|eBdB ~A3G|Bdef gfed|
e~A3 BGEF|G2BG DG (3ABd|eBdB AAGA|[1Beed e3d:|
On Rakudo, there were no error messages, but “A Hat for the Whales” was the last tune processed, even though there were a dozen more tunes in the ABC file after it.
Without thinking about it too much I corrected this batch of errors. Now on Niecza the entire file parsed, but I got the error
Unhandled exception: Illegal key signature B minor
Under Rakudo, the same ABC file just appeared to work.
What’s going on here? Well, key signature processing looks something like this (simplified):
token key-def { <basenote> <chord_accidental>? <mode>? }
token mode { <minor> | <major> }
token minor { "m" ["in" ["o" ["r"]?]?]? }
token major { "maj" ["o" ["r"]?]? }
No space allowed between
basenote and
mode. So when you invoke parse with that rule on “B minor”, in Niecza it fails, because the entire string was not successfully parsed. In Rakudo it succeeds, because “B” is a valid key signature, and it doesn’t care that the entire string was not parsed. In practice, this means that three of the tunes in the collection got the wrong key signature: B major instead of the correct B minor.
To my mind, Niecza’s behavior is more intuitive. If you’re trying to parse a file, parsing half of it and then silently ignoring the rest is wildly unhelpful behavior. In every use I’ve had for parse so far, I’ve wanted it to parse everything in the string I sent to the function.
But hey, I understand that some people might have different needs. Can we at least add a flag that will give you a parsing failure if the entire string is not parsed? :all, perhaps?
March 06, 2013 12:59
March 02, 2013
I would put a description, but who seriously cares?
New features
Perl 6 specification
- Inside of
try is considered eager context, unless try is called in sink context, when it’s in sink context.
Rakudo Perl
- Type placeholders (like
Int, which is null, except Int typed) now have their .gist method show their name within parens to suggest being undefined. For example, (Int), (Mu).
- Changed “Cannot assign to a non-container” error message to “Cannot modify an immutable value” that hopefully is more clear.
- Release announcements now use Markdown.
Cool.substr doesn’t coerce its first argument to Int anymore.
March 02, 2013 08:00
March 01, 2013
So, well, I have a VPS, and decided to put a webpage on it for lulz. You can see it at mango.uk.to, and the source (sauce) is available on my GitHub profile.
Just mentioning, not that this site is important. I just wanted to run a web server just to run a web server.
By the way, this blog is also open sauce, you can see its source code on GitHub. It’s full of hacks, but that’s Jekyll for you.
March 01, 2013 08:00
February 24, 2013
On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the February 2013 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the February 2013 release is available from http://rakudo.org/downloads/star/. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.
In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2013.02.1 of the Rakudo Perl 6 compiler, version 4.10.0 of the Parrot Virtual Machine, and various modules, documentation, and other resources collected from the Perl 6 community.
Some of the new features added to this release include:
- “Did you mean …” suggestions for symbol-not-found errors
- Compile-time optimization of some cases of junctions in boolean context
- IO::Socket.get now works again with non-ASCII characters
- constant folding for routines marked as ‘is pure’
- natively typed variables and better error reporting in the REPL
- speed up eqv-comparison of Bufs
- warnings for useless use of (some) literals, variables and constant expressions in sink context
This release also contains a range of bug fixes, improvements to error reporting
and better failure modes.
The following features have been deprecated or modified from previous
releases due to changes in the Perl 6 specification, and are being removed
or changed as follows:
- .gist on a type object will return ‘(Typename)’ instead of ‘Typename()’. If you want to get the class name alone, continue to use $obj.^name
- postcircumfix:<[ ]> and postcircumfix:<{ }> will become multi-subs rather than multi-methods. Both at_pos and at_key will remain methods.
- Unary hyper ops currently descend into nested arrays and hashes. This will change to make them equivalent to a one-level map.
- The Str.ucfirst builtin is deprecated; it will be replaced by Str.tc.
- Leading whitespace in rules and under :sigspace will no longer be converted to <.ws> . For existing regexes that expect this conversion, add a <?> in front of leading whitespace to make it meta again.
- The ?-quantifier on captures in regexes currently binds the capture slot to a List containing either zero or one Match objects; i.e., it is equivalent to “** 0..1″. In the future, the ?-quantifier will bind the slot directly to a captured Match or to Nil. Existing code can manage the transition by changing existing ?-quantifiers to use “** 0..1″, which will continue to return a List of matches.
There are some key features of Perl 6 that Rakudo Star does not
yet handle appropriately, although they will appear in upcoming
releases. Some of the not-quite-there features include:
- advanced macros
- threads and concurrency
- Unicode strings at levels other than codepoints
- interactive readline that understands Unicode
- non-blocking I/O
- much of Synopsis 9
There is an online resource at http://perl6.org/compilers/features
that lists the known implemented and missing features of Rakudo
and other Perl 6 implementations.
In many places we’ve tried to make Rakudo smart enough to inform the
programmer that a given feature isn’t implemented, but there are
many that we’ve missed. Bug reports about missing and broken
features are welcomed at
href="mailto:rakudobug@perl.org">rakudobug@perl.org.
See perl6.org for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. A draft of a Perl 6 book is available as UsingPerl6-draft.pdf> in the release tarball.
The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.
February 24, 2013 19:21
February 23, 2013
Now, I’m back to the your regularly scheduled Perl 6 changes article that isn’t. Well, it is, but it’s not. Or something. Rakudo 2013.02.1 was released (well, 2013.02 was released too). The version bump was caused by the fact that NQP 2013.02 had broken native call support.
Incompatible changes
Perl 6 specification
- Failed matches result
Nil, not (Match).
New features
Perl 6 specification
- Removed
Set.powerset. It was removed after 10 hours of existing. Consider it a printing error in my previous article, or something. Not really an incompatible change, as it practically never existed.
- Type placeholders (like
Int, which is null, except Int typed) now have their .gist method show their name within parens to suggest being undefined. For example, (Int), (Mu).
STD.pm6
- Added a warning for smartmatch of
(Match) or (Nil).
Rakudo Perl
- The sink context now can be detected. Side-effect-free operations, constants and variables warn about it.
- Compiler warns about apparent subscripts in regexes being treated as regexes (for example,
@a[0], which is treated as any of @a elements followed by 0).
- It’s possible to interpolate arrays in regular expressions.
- Added
int64 native type.
/./ and rx/./ now matches against $_ in sink context.$feature is not yet implemented. Sorry. error message doesn’t have is anymore.- Coercive type declarations make not yet implemented error.
Niecza Perl
- Type placeholders (like
Int, which is null, except Int typed) now have their .gist method show their name within parens to suggest being undefined. For example, (Int), (Mu).
February 23, 2013 08:00
February 22, 2013
<grondilu> masak: are you sure problem #1 is computationnaly doable? It's not NP = P or something, is it?
* grondilu thinks of it and realizes the number of possibilities is not so large
<grondilu> .oO( 2**number_of_islanders possibilities anyway )
And so, the time-honored tradition of reviewing p6cc solutions begins. Again.
For our first task, we have the very human conundrum of figuring out who is telling the truth and who is lying. Fortunately, on the island of Smul, that's a much more tractable problem than in the real world.
The problem description follows.
## Tell knights from knaves based on what they say
On the mythical island of Smul, people suffer from a rare genetic disorder that
make them either tell the truth all the time, or lie all the time. These are
the only two types of people on the island, known as knights and knaves,
respectively.
Write a program that takes as input a number of utterances by islanders, and
outputs for each person whether that person is a knight or a knave. If there is
no possible assignment that works, the program should report that no solution
exists. In the case of multiple solutions, the program should report every
possible solution.
The islanders can make four different classes of utterances:
X is a knight.
X is a knave.
X and Y are of the same type.
X and Y are of different types.
(Here, X and Y are used as metavariables, of course, and can in fact be any
name of an islander.)
Islanders can refer to each other. The same islander can make several
utterances. If an islander mentions another islander that doesn't say anything,
your program should consider the entire input to be erroneous.
Here are a few examples:
A: A is a knight.
Both a knight and a knave would assert the same thing. So this input has two
solutions.
B: B is a knave.
Neither a knight or a knave would ever say this about themselves. So this input
allows no solution.
C: C and D are of the same type.
D: D and C are of different types.
Here, the two islanders are contradicting each other, so one of them must be a
knight and the other a knave. But this is exactly what D is saying, so D is the
knight. One solution.
Let's get one obvious thing out of the way: we can solve this with a brute-force method, supposing for each islander in a situation first that he is a knight and then that he is a knave. Each islander thus bifurcates the universe in two possible universes. Two islanders will yield four possible universes to investigate. Three islanders will yield eight universes. Ten islanders will yield 1024 universes. The universes grow exponentially with the islanders. The brute-force solution will always work, but at an exponential slowdown.
Many people choose the brute-force solution. I don't blame them; it's there for the picking. Some people get fancy with logical propositions or "affiliations" between islanders, but they all fall down the exponential pit at one point or another. The brute-forcers are all over the place in terms of style and brevity, and it's quite fun to watch.
Then there's this one contestant that gets the nice solution. It always thrills me when that happens.
Here, let me lay it out for you. Let 0 mean False/knave and 1 mean True/knight. Then let's translate the four possible utterances to equations:
- "A: X is a knight." → A = X
- "A: X is a knave." → A = 1 xor X
- "A: X and Y are of the same type." → A = 1 xor X xor Y
- "A: X and Y are of different types." → A = X xor Y
By some amazing coincidence, all the utterances can be put into more or less the same mold, and the only operator used is xor. It's actually fun to translate these formulas back, and get alternative formulations of things, sometimes giving a different perspective on things:
- A = X → "A is as truthful as X"
- A = 1 xor X → "A or X is truthful, but not both"
From that first formulation we see why, when A is asserting that A is a knight, that utterance devolves into a tautology A = A. It's as if an islander cannot assert his own knighthood solely on his own authority.
From the second formulation we see why, when A is asserting that A is a knave, such an utterance devolves into a contradiction A = 1 xor A. There just isn't any such number A. (We'd have to look in the murky domain of fuzzy logic, but that's outside the realm of the island of Smul.)
Anyway, translating all the utterances to this form is a big win. Now we can make a linear equation system of all the utterances, and solve the linear equation system. Do we have fast algorithms for that? Yes, we do! Gaussian elimination has an arithmetic complexity of O(n3). That's quite an improvement on exponential brute force!
We must take care to do all the additions and subtractions as xors, though. This is because our underlying algebra is truth values, outside of which we may not stray. So we're really doing Gaussian elimination on the quotient ring Z/2Z.
In the literature, we find this problem as XOR-satisfiability. Wikipedia dignifies it with two sentences.
One contestant did it this way. Yay him. You should check out his solution.
Here are the solutions, and my reviews of same. Enjoy.
Next up: rectangle haikus, possibly the most fun I've ever had with a p6cc task.
February 22, 2013 22:39
I’m not sure if anybody noticed, but lately, nothing was happening on my blog. So, I’m going to write an article about friendly interactive shell (fish), the best shell I’ve used ever (it’s my personal opinion, in case you haven’t noticed, see my footer blog). Actually, I think I’ve moved to it half of the year ago or something.
These days, the two popular shells exist - bash and Z shell. I’m aware of that more shells exist - just nobody uses them these days By the way, if you try to find logical fallacy, I agree, cmd.exe and Windows PowerShell are often used, but those aren’t UNIX shells. Oh, and programming language REPLs are often used too (like irb), but they aren’t UNIX shells too.
Bash
The bash shell is standard shell, included in many Linux distributions. It will make you bash your head against the brick wall (pun intended).
One of examples of bash code I could have found is this. And trust me, it’s not the worst thing (thanks, Wikibooks).
# print the powers of two, from 1 to 512:
for ((i = 1; i < 1000; i *= 2)); do
echo $i
done
Do you see double parens here? This is really strange math syntax that only supports integers. Why double parenthesis? Well, the answer is compatibility with Bourne shell (really, really old shell, usually called sh (but sometimes sh is actually bash, to make you bash)).
Also, you could have noticed keywords do and done. Bash could easily not require do keyword before for loop body. Yet, it’s required. The other strange quirks is that you aren’t allowed to put semicolon after do keyword. Why? I’ve no idea, really. The done word, while I think it’s needed, differs between blocks. For example, if blocks end with fi.
Bourne again shell has lots of strange syntax you really wouldn’t ever use. For example, bash has $'' syntax. It unescapes escape sequences inside. The strange part is '' in bash doesn’t do any processing at all, yet $'' does. Bash also has $"" syntax, but it doesn’t do escape processing - instead it uses gettext to translate string inside.
# Prints 'Hello world.', with new line in middle.
echo Hello$'\n'world.
# Prints 'Hello\nworld.', literally.
echo Hello$"\n"world.
Consistency in bash is even worse than in languages such as PHP. There are many ways to do something, but none of them are obvious. And I mean that, none. Expanding bash is adding new syntax that totally doesn’t make sense, but was a syntax error in previous versions. $ sign is used for many features that have nothing with variables.
Also, bash integrates many programs in it that really shouldn’t be inside. For example, echo.
[glitchmr@pineapple ~]$ echo --help
--help
[glitchmr@pineapple ~]$ which echo
/bin/echo
[glitchmr@pineapple ~]$ /bin/echo --help
Usage: /bin/echo [SHORT-OPTION]... [STRING]...
or: /bin/echo LONG-OPTION
Echo the STRING(s) to standard output.
...
NOTE: your shell may have its own version of echo, which usually supersedes
the version described here. Please refer to your shell's documentation
for details about the options it supports.
Report echo bugs to bug-coreutils@gnu.org
GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
Report echo translation bugs to <http://translationproject.org/team/>
For complete documentation, run: info coreutils 'echo invocation'
The /bin/echo on my system is GNU echo. The bash on my system is GNU bash. They were both made by GNU. Yet, they act differently. Is there a better recipe for failure than program that actually isn’t called because your shell is “clever”.
It’s not just echo. Many commands are implemented in shell, when they should be implemented outside shell. For example, pwd. Why exactly? To confuse you when dealing with recursive structures.
[glitchmr@pineapple ~]$ mkdir recursion
[glitchmr@pineapple ~]$ cd recursion/
[glitchmr@pineapple recursion]$ ln -s . recursion
[glitchmr@pineapple recursion]$ cd recursion
[glitchmr@pineapple recursion]$ pwd
/home/glitchmr/recursion/recursion
[glitchmr@pineapple recursion]$ which pwd
/bin/pwd
[glitchmr@pineapple recursion]$ /bin/pwd
/home/glitchmr/recursion
[glitchmr@pineapple recursion]$
The shell outright lies about your location. You’re in /home/glitchmr/recursion. Why shell does that? Well, it does that to not confuse you when typing cd ... And indeed, if you type cd recursion/recursion, and cd .. after that, you will end in recursion directory, not the directory where you created the first recursion directory.
Symbolic links in UNIX are simply redirects. If you made a symbolic link to /usr/share/hell, you shouldn’t expect that cd .. made after following link follow return you anywhere that isn’t /usr/share. Yet, implementation of symbolic links in bash lies.
The lie is obvious after using anything that isn’t bash builtin. For example, ls .. will always show contents of /usr/share, even if you made symbolic link in /tmp/heaven. Yet, cd .. will return to /tmp/heaven.
By the way, in case you ask, cd is also a builtin. But that’s acceptable, because external command cannot change environment of caller (such at PATH). So it has to be a builtin. Yet, somehow, some people are still confused.
Z shell
Z shell is a shell that is way better than bash. Well, if you configure it properly. It’s a shell that appears to have some compatibility with Bash scripters (for example $'' syntax works, but $"" doesn’t (or rather, it returns literal ’$’ and contents of "")).
Z shell has lots of features. In fact, the author is “not aware of a major interactive feature in any other freely-available shell which zsh does not also have (except smallness)”. Something that would be sane default actually isn’t default. In fact, with default settings it mostly works like bash.
For example, completion. By default, Z shell only completes file names and command names. If you will type sudo apt-get install, press space and then press Tab, it will give you file names. Useful? I don’t think so.
Of course, the option to enable better Tab completion is possible to turn on. It’s just not default. Why the shell would have default that don’t make sense. Simiarly, by default Tab completion cannot expand globs. Yet, it’s possible to enable.
When you start Z shell for first time, you see the configuration wizard (called zsh-newuser-install). The problem is, it’s the most non-user-friendly thing I’ve seen. It asks about minor details you don’t care about, without proposing any useful default. For example, the maximum number of errors you can make for the command to be still proposed in error message. I just have pressed Enter, and the response is “Please enter a number”. Do you think I care about this?
Actually, as far I know, nobody actually uses this wizard. In most cases, zsh is configured using so called oh-my-zsh. What it is actually? Well, it’s a config script with sane things configured by default. Yet, it has got 8,648 stars on GitHub and 3,326 forks as I’m writing it.
Yes, it’s just ZSH config file that has things that should be default, but they aren’t. Oh, and plugins array that contains list of plugins that mostly make aliases and completion adders that aren’t in main zsh (like CoffeeScript). For example git plugin adds aliases like gc for git commit -v.
I don’t know about you, but I don’t like programs which take 4 hours to configure. Z shell is one of these. oh-my-zsh makes the job easier, but I still don’t think it’s worth wasting the time. Perhaps if fish wouldn’t exist, it would be worth it.
friendly interactive shell
Z shell, aside of totally insane configuration isn’t a bad shell (if I would want to talk about bad shells, it would be csh or cmd.exe). friendly interactive shell is as good as Z shell (with good Z shell configuration), except without having to configure it.
In fact, most of things cannot be configured. You cannot configure nonsense like maximal history length (by design). You cannot configure interpretation of numbers starting with 0 as octal (you can in Z shell - I have no idea why Z shell even allows you to type “echo 010” to get “8” when configured to do so - but it does). You cannot disable file globs.
Remember .bashrc/.zshrc configuration file? You won’t need it. The colors and themes can be easily configured using a web service (but if you prefer, you can use set command directly).
For example, if you want to configure your EDITOR to be vim, you can run the following command.
set -Ux EDITOR vim
The -U modifier means that this value is stored between session. The -x means that this variable is exported to programs ran by shell. After you have done that, your variable magically appears in every fish session.
Adding new functions is also easy. It’s enough to type funced function-name. After
Fish also offers features I haven’t seen in other shells, like syntax highlighting. Ommited quote? Syntax highlighting will help you.
In fish, everything is tab-completeable. Even if the program doesn’t have specific tab completion definitions, it’s still tab-completeable, thanks to generating tab completions from man pages. For example, gedit doesn’t have specific tab completions, yet fish managed to create them.
glitchmr@pineapple ~> gedit -
--background (Run gedit in the background.)
--encoding (Set the character encoding to be used for openi… [See Man Page])
--geometry (Set the X geometry window size (WIDTHxHEIGHT+X+Y).)
--help (Prints the command line options.)
--list-encodings (Display list of possible values for the encodin… [See Man Page])
--new-document (Create a new document in an existing instance of gedit.)
--new-window (Create a new toplevel window in an existing instance of gedit.)
--standalone (Run gedit in standalone mode.)
--version (Output version information and exit.)
--wait (Open files and block the gedit process.)
-b (Run gedit in the background.)
-g (Set the X geometry window size (WIDTHxHEIGHT+X+Y).)
-s (Run gedit in standalone mode.)
-w (Open files and block the gedit process.)
Of course, just like zsh (with correct configuration), it can expand more stuff. For example, kill command expands process IDs.
glitchmr@pineapple ~> kill
1 (systemd) 127 (systemd-udevd) 479 (gconfd-2)
2 (kthreadd) 132 (systemd-journal) 508 (gvfsd)
3 (ksoftirqd/0) 229 (hd-audio0) 512 (gvfsd-fuse)
5 (kworker/0:0H) 230 (kpsmoused) 527 (cinnamon [tty1])
7 (kworker/u:0H) 299 (irq/46-mei) 528 (colord)
8 (migration/0) 307 (cfg80211) 534 (polkit-gnome-au [tty1])
9 (rcu_preempt) 308 (hci0) 536 (dropbox [tty1])
10 (rcu_bh) 309 (kworker/u:1H) 539 (nm-applet [tty1])
11 (rcu_sched) 310 (scsi_eh_6) 540 (gnome-screensav [tty1])
12 (watchdog/0) 311 (rts5139-control) 558 (gsd-printer [tty1])
13 (watchdog/1) 312 (rts5139-polling) 566 (gvfs-udisks2-vo)
14 (ksoftirqd/1) 314 (ttm_swap) 568 (udisksd)
15 (migration/1) 325 (NetworkManager) 598 (gvfsd-trash)
17 (kworker/1:0H) 329 (dbus-daemon) 728 (firefox [tty1])
18 (cpuset) 331 (systemd-logind) 744 (hexchat [tty1])
19 (khelper) 332 (login) 746 (pxgsettings [tty1])
20 (kdevtmpfs) 335 (polkitd) 797 (dconf-service)
21 (netns) 341 (mysqld) 798 (gnome-terminal [tty1])
22 (bdi-default) 344 (wpa_supplicant) 805 (gnome-pty-helpe [tty1])
23 (kblockd) 348 (nginx) 806 (fish [pts/0])
26 (khungtaskd) 349 (nginx) 1374 (kworker/1:2)
27 (kswapd0) 351 (php-fpm) 5790 (fish [pts/1])
28 (ksmd) 352 (php-fpm) 10502 (gvfsd-metadata)
29 (khugepaged) 353 (php-fpm) 13551 (fish [pts/2])
30 (fsnotify_mark) 373 (dhclient) 13635 (ssh [pts/2])
31 (crypto) 385 (fish [tty1]) 13866 (kworker/1:0)
35 (kthrotld) 387 (fishd) 18892 (gvfsd-http)
38 (deferwq) 412 (startx [tty1]) 20044 (fish [pts/1])
62 (khubd) 429 (xinit [tty1]) 20046 (python [pts/1])
63 (ata_sff) 430 (X) 20405 (vim [pts/0])
64 (scsi_eh_0) 434 (gnome-session [tty1]) 21370 (kworker/0:1)
65 (scsi_eh_1) 437 (dbus-launch [tty1]) 21373 (kworker/u:2)
66 (scsi_eh_2) 438 (dbus-daemon) 22399 (fish [pts/3])
67 (scsi_eh_3) 440 (at-spi-bus-laun) 22472 (vim [pts/3])
68 (scsi_eh_4) 444 (dbus-daemon) 22521 (kworker/u:0)
69 (scsi_eh_5) 447 (at-spi2-registr) 22863 (kworker/0:0)
93 (kworker/0:1H) 455 (gnome-settings- [tty1]) 22945 (kworker/u:1)
95 (kworker/1:1H) 464 (pulseaudio) 23238 (ps [pts/4])
103 (jbd2/sda6-8) 465 (rtkit-daemon) 23239 (grep [pts/4])
104 (ext4-dio-unwrit) 468 (upowerd) 23240 (tail [pts/4])
110 (flush-8:0) 476 (gconf-helper)
Of course, fish has some disadvantages. One of disadvantages is the syntax. It’s not POSIX compatible by design. For example, the history substition doesn’t exist. Instead, you’re supposed (if you wanted, let’s say, sudo) to press Up, Home, and type ‘sudo’. Command substitution uses (), not `` or $(). Still, if you will accidentally use $() bashism, the shell will inform you with an error.
Also, the newest stable is really old and slow. Instead I would use version from git, which is usually more stable than stable version. In Arch Linux, it’s fish-shell-git from AUR, on other operating systems (like CentOS I use on my VPS), you have to compile it yourself. It isn’t so difficult. The GitHub repository is fish-shell/fish-shell.
As for why I’m mentioning fish shell, well, I have lately contributed a small patch to hint -C suboptions in perl - like -CSDL. Nothing special, but hey, why not make a blog post, considering I write them rarely.
February 22, 2013 08:00
February 20, 2013
Flat on the bunk again, he ran for his life. The Parrot stalked him
through the grey hours of morning, smoothing its fractal feathers,
shuffling itself slowly into clarity as though at the end of a
flashy film-dissolve, until at last his mind's eye had to acknowledge
a shape,
a shape,
a wink
-- From BLIT, a short story by David Langford
http://www.infinityplus.co.uk/stories/blit.htm
On behalf of the Parrot team, I'm proud to announce Parrot 5.1.0, also known
as "Zombie Parrot". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.
Parrot 5.1.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/supported/5.1.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.
Parrot 5.1.0 News:
- Core
+ The .sort() method was added to the FixedFloatArray PMC
+ Improved detection of system memory for machines with >2GB
+ Improved pbc_to_exe support for spacey paths
+ Fixed Parrot_io_readall_s allocating too much string space
- Build
+ Fixed generated MANIFEST files to omit $destdir
- Documentation
- Tests
+ .readall now checks that prior reads are respected.
- Community
+ Weekly IRC meetings have resumed. #parrotsketch Tuesdays at 1930 UTC
The SHA256 message digests for the downloadable tarballs are:
af26c2fcc806505ec516ebb013bdd37b218633f5fe63faaa6b843ffe55e0135e parrot-5.1.0.tar.bz2
2483963c1bec665be772cb40a71fd3d9d2621feca547932475017c81a2f7e49b parrot-5.1.0.tar.gz
Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 19 Mar 2013.
Enjoy!
--
Bruce Gray (Util of PerlMonks)
February 20, 2013 18:36
February 16, 2013
Having just reached an interesting milestone, I thought I’d blog a quick progress update on my work to port NQP to the JVM, in preparation for also doing a port of Rakudo Perl 6.
The big news is that the grammar and regex engine is pretty much ported. This was a sizable and interesting task, and while a few loose ends need to be wrapped up I think it’s fair to say that the hard bits are done. The port includes support for the basics (literals, quantifiers, character classes, positional and named captures, calls to other rules, and so forth) as well as the more advanced features (transitive Longest Token Matching for both alternations and protoregexes, and embedded code blocks and assertions). Missing are anchors, conjunctions and a small number of built-in rules; none of these are particularly demanding or require primitives that don’t already exist, however. It’s also worth pointing out that the NQP code to calculate the NFAs used in Longest Token Matching also runs just fine atop of the JVM.
Another interesting feature that I ported a little while ago is multiple dispatch. This was some effort to port, since the original implementation had been done in C. While it’s sensible to have a close-to-the-VM dispatch cache, there’s little reason for the one-off candidate sorting work (not a hot path) to be done in C, so I ported the code for this to NQP. This meant that on the JVM side, I just needed to implement a few extra primitives, and could then run the exact same candidate sorting code.
I think it’s worth noting again that I’m really doing two things in parallel here: hunting down places where NQP couples too tightly to Parrot and loosening the coupling, and also doing the porting to the JVM. The first half of this work is relevant to all future ports. In many cases, I’m also finding that the changes give us architectural improvements or just cleaner, more maintainable code. I wanted to point this out especially because I’m seeing various comments popping up suggesting that Rakudo (or even Perl 6) is on a one-way road to the JVM, forsaking all other platforms. That’s not the case. The JVM has both strengths (mature, a solid threading story, widely deployed, the only allowed deployment platform in some development shops, increasing attention to supporting non-Java languages through things like invokedynamic) as well as weaknesses (slow startup time, lack of native co-routine support, and the fact that it was originally aimed at static languages). Rakudo most certainly should run on JVM – and it most certainly should run on other platforms too. And, as I wrote in my previous post, we’ve designed things so that we are able to do so. Perl has always been a language where There’s More Than One Way To Do It. Perl also has a history of running on a very wide range of platforms. Perl 6 should continue down this track – but the new reality is that a bunch of the interesting platforms are virtual, not hardware/OS ones.
By now, the JVM porting work is fast approaching a turning point. Up until now, it’s been about getting a cross-compiler and runtime support in place and working our way through the NQP test suite. This phase is largely over. The next phase is about getting NQP itself cross-compiled – that is, cross-compiling the compiler, so that we have an NQP compiler that runs on the JVM, supporting eval and able to run independently.
February 16, 2013 23:16
Today Perl 6 is as old as Perl was when Perl 6 was announced.
$ perl6 -e 'say Date.new(2000, 7, 18) + (Date.new(2000, 7, 18) - Date.new(1987, 12, 18))'
2013-02-16
The 1987 date is the release of Perl 1. The 2000 date is the throwing of the mugs, which I consider to be Perl 6's birthday.
It's a bit interesting to compare Perl 6 at this point with Perl back then. They are two fairly different projects, even though the people are overlapping to a great extent.
Bottom-up vs top-down
In any programming project, you can start from the small pieces and build upwards to the overreaching goals and ideas, or you can start from the ideas and build downwards to the nitty-gritty stuff.
Perl is a bottom-up project: the Perl 1 release looks puny today. It didn't do much. But it did run. It did solve people's problems. And there's an unbroken chain of commits leading from Perl 1 to today's Perl 5.
Perl 6 is definitely a top-down project. Larry mulled over those RFC's and wrote the Apocalypses. These eventually resulted in the Synopses, which guide implementation work. Implementations reach up towards the spec, and are in some sense always built bottom-up... but on the most zoomed-out scale, Perl 6 is built from the top down.
This difference was very much deliberate. For the Perl 6 project, it was felt that a specification (and a corresponding test suite) was needed. The Pugs project is currently very dormant and not actively developed, but it did result in both the Synopses and the spectest suite. Both are essential artifacts for any Perl 6 implementer.
Second system syndrome
"It's important to remember that when you start from scratch there is absolutely no reason to believe that you are going to do a better job than you did the first time." — Joel Spolsky, Things You Should Never Do, Part I
Look, here's the thing: everyone knows that The Big Rewrite isn't a good idea. That's like, Software Project Management 101. Even the people going into this project knew that. The jokes about second systems have been with us from the start, as a part of our cultural heritage.
There have been efforts to mitigate the risks and to calm people on the way. The Ponie project was an effort to put Perl 5 on Parrot, for example, to provide for a migration path or simply a communication bridge from Perl 5 to Perl 6.
Early on, it was also felt that Perl 6 wouldn't be so different from Perl 5, syntactically. Sure, the sigils would come out being invariant, and a comma would have to be added here and there for consistency. But that was about it. Perl 5 programmers would still feel right at home.
Well, guess what? The Ponie project, even with brilliant developers behind it, slowed and finally halted at the height of the Pugs era. Turns out the reasons people wanted to further themselves from the existing Perl 5 internals — that they are a big hairy mess of intertwined C macros — also made porting to Parrot exceedingly difficult. Ponie brought some permanent improvements to the Perl 5 core, but in the end it didn't reach its goal of Perl 5 targeting Parrot.
In the meantime, Perl 6 kept improving and evolving, syntactically as well as semantically. The differences piled up. Perl 6, unfettered by backward-compatibility, could take sometimes vast leaps and reach places in the state-space Perl 5 could only dream of. Also, things kept shaking around and stabilizing, features growing together into a unified whole instead of this. All good and well, but it meant that Perl 6 drifted further away from Perl 5.
So here we are, a decade later, with two distinct languages, and no way for them to interoperate. Having them actually talk to each other is still very much on the agenda. It's just that it's a big undertaking. A couple of recent projects are attempting to put Perl 5 on a platform where it can talk to Perl 6.
I think the biggest thing to realize in all this is that Perl 6, even from the very start, could never have followed the trajectory Perl did. There was one thing that existed in 2000 that didn't exist in 1987, when Perl was announced: Perl. That's why Perl 6 is a top-down project. That's why it's a second system. And that's why the obvious measure-stick of Perl 6 is Perl, with its 13-year head start.
Rubber, meet road
I'm not complaining about this, mind. We should compare Perl 6 to Perl. And to all other scripting languages that have popped up in the same niche. We should steal ideas and adapt to modern practices, like we always do, in both Perl and Perl 6.
But for a top-down project, the biggest challenge is always to reach all the way down to the ground: to actually start being useful for someone. That's perhaps our big challenge. It was back in 2008 when I became heavily involved in the Perl 6 effort, and it still is today: be useful. Be usable. Put food on someone's family.
Perl 6 is useful to me today. Not to the extent that I can write Perl 6 code all day, but to the extent that it makes me more effective and more productive now and then. Bringing that kind of usefulness to others requires lots of work writing modules, documentation, books, and tutorials, as has been discussed elsewhere. We're making some headway with this. I'm more optimistic than I was two years ago.
We also need to work on things such as performance. Perl 6 is fast enough for some things, but overall implementations are still fairly slow. Sometimes ridiculously slow. Good work is being done in this area, too.
But here's what makes me the most optimistic about the Perl 6 effort: after a few years of watching things evolve, I've noticed that while Perl 6 is being developed top-down on the outermost scale, it's actually a series of bottom-up projects that drive Perl 6 forwards.
jnthn likes to tell about how he promised to implement junctions in Rakudo back in 2008, and then realized that junctions were actually tied to method dispatch and the object system, so he had to implement those, too. Later, while pmichaud was rebuilding the parser according to our current understanding of it, jnthn was building Rakudo's meta-object protocol, a project called 6model. Each step on the way replacing the layer below based on what we had found that the layer above required. The 6model work is part of what now enables us to port nqp and Rakudo to the JVM.
That's what makes me optimistic. While Perl 6 is undeniably, unchangeably a top-down project, highly competent people are factoring that top-down knowledge into the design of components in a bottom-up way. What's been happening in these 13 years is that we've become increasingly better and more efficient at building Perl 6.
So, how're we doing?
"Chuck Norris has actually been using Perl 6 since 1987, and has been waiting for Larry to play catch-up. :)" — dukeleto on #perl6
Let's stop and compare the state of Perl 6 today with the state of Perl back in 2000.
Mugs. Back then, Jon Orwant threw coffee mugs against a wall because he felt there was nothing to energize the community and people would walk off to other things.
Are sixers doing any mug-throwing nowadays? Yup, we did, at the Perl Reunification Summit. Liz felt, and still feels, that if we don't actively bring Perl 5 and Perl 6 together, then both projects are eventually going to fizzle out.
Crisis. As part of that, the feeling Jon Orwant was expressing was that Perl was stagnating or going nowhere.
Is Perl 6 stagnating? I wouldn't say so. We're still extremely small compared to Perl 5, but my general sense is that we're still gaining speed, becoming stronger every day. Still very much pre-peak.
Usefulness. In 2000, Perl was on its thirteenth year of solving everyday problems for people.
What about Perl 6 today? Well, it's very different. Part of this is due to the top-down nature of the project. People are waiting for it to be useful to them. Part of it is perception: people are waiting, instead of jumping in and using the language in cases where it could already be useful to them. We see this in the verb tenses people are using when they ask stuff: "Will Perl 6..." Perl 6 is about the future, but if we don't do something about that perception, we'll never get there. The problem with "tomorrow" is that it never actually arrives.
Community. In 2000, Perl drew thousands of people to conferences each year.
Is Perl 6 drawing thousands? No. The Perl people are kind enough to let us piggyback on Perl conferences and workshops, and through them we have an audience of thousands. I keep feeling that there's a big interest in Perl 6, both from within the Perl community and from other communities. But there's also a sense of waiting, of not-yet-production-ready. (And I agree.)
What remains?
Which brings us to the last, perhaps most important question. What's left for Perl 6 to become a viable, useful solution to most people out there?
For years I wouldn't go near that question. Just working along, head looking down at the current sub-projects, making Perl 6 more useful to myself and hoping that was enough. But my recent visit to FOSDEM made me realize that some kind of production release is actually within reach — say, a few years away — and a focused effort to reach a releasable state by some criteria would be highly useful for us and for others.
So here are my for criteria. This is what Perl 6 needs to be ready for the world.
Features. Slightly simplified, this is what we've been working on since the Pugs days. We are doing well on features. There's not much I feel is missing nowadays from the core language. It's a pleasant language to use and to express thoughts in. There are bugs, sure, but with each month that passes, those bugs are becoming less frequent and less serious.
Speed. Working on it. I won't make any promises about the JVM port, but I have a strong feeling it will make a big difference. Parallel to this, there are various already-implemented or to-be-implemented efforts to make the compiler smarter about generating efficient code.
Concurrency. There are some good ideas in the specification about concurrency. Some of them will work fine, some probably won't pan out. What's needed now is a solid implementation of concurrency in either Rakudo or Niecza, a test-bed for the ideas in the spec, so that users can get a feel for what's there and what's missing. I'd really like for Perl 6 to have a decent concurrency story, because it's one of the things that were promised early on for Perl 6, and one of the things that Perl never got right. Also, concurrency in all its various forms seems to be growing increasingly important in the programming world of today.
CPAN. Sure, we have an ecosystem for Perl 6. But it isn't CPAN. It's three orders of magnitude as small as CPAN. And it's not realistic to port all of CPAN either — that'll simply never happen. What we really need is a high-bandwidth connection between Perl 5 and Perl 6. We need for them to run in the same runloop. Once they do that, Perl 6 programmers can have their CPAN modules, and Perl 5 programmers can work Perl 6 code into their projects. This is also something that was promised early on. It's not an easy task.
Conclusion
Perl 6 is a multi-year project. Though today's date is a little bit arbitrary, it's worthwhile to look both backwards and forwards, to see where we currently are. And though we're structurally different from the Perl project, it's still interesting to make comparisons.
We believe we're building something really nice with Perl 6. 2013 may not be the year when we're finally production-ready, but it sure feels like a year where a lot of significant things will happen (and are already happening). And, unlike 2012, I finally feel ready to speculate about the light at the end of the tunnel.
February 16, 2013 18:00
> my $perl = Date.new: '1987-12-18' # Perl 1 release[1]
1987-12-18
> my $perl6 = Date.new: '2000-07-18' # Perl 6 announcement[2]
2000-07-18
> Date.today - $perl6
4596
> $perl6 - $perl
4596
Normally, I wouldn’t put irrelevant snippets of code[citation
needed]. But this day is different - we have Perl 6 equidieversary (thanks, moritz). It’s 4596 days since Perl 6 was announced (Wikipedia says it was annouced 2000-07-19, but actually it was earlier). In 2000-07-19, Perl was 4596 days old. But I won’t discuss about this more, as masak already made perfectly fine article about that.
Let’s return to your regularly scheduled program.
New features
Perl 6 specification
-
Sets and Bags do not flatten in list context.
-
Metaoperators are treated as normal operators, and aren’t making list of closures. So, for example @array X* * is -> $arg { @array X* $arg }.
-
Inner classes and roles must be lexically scoped.
-
Added List.combinations and Set.combinations.
-
Added Set.powerset. It’s defined like that.
multi method powerset { set ∅, self.combinations }
Rakudo Perl
$_ is writable in -n/-p code (one-liner).eqv is faster for Buf objects.- Configure script fails on unknown command line options.
February 16, 2013 08:00
February 13, 2013
When talking about pattern matching in the context of Perl 6,
people usually think about regex or grammars. Those are indeed very powerful
tools for pattern matching, but not the only one.
Another powerful tool for pattern matching and for unpacking data
structures uses signatures.
Signatures are "just" argument lists:
sub repeat(Str $s, Int $count) {
return $s x $count
}
Nearly all modern programming languages have signatures, so you might say:
nothing special, move along. But there are two features that make them more useful
than signatures in other languages.
The first is multi dispatch, which allows you to write several
routines with the name, but with different signatures. While extremely
powerful and helpful, I don't want to dwell on them. Look at Chapter 6 of
the "Using
Perl 6" book for more details.
The second feature is sub-signatures. It allows you to write a
signature for a sigle parameter.
Which sounds pretty boring at first, but for example it
allows you to do declarative validation of data structures. Perl 6 has no
built-in type for an array where each slot must be of a specific but different
type. But you can still check for that in a sub-signature
sub f(@array [Int, Str]) {
say @array.join: ', ';
}
f [42, 'str'];
f [42, 23];
Here we have a parameter called @array, and it is followed by
a square brackets, which introduce a sub-signature for an array. When calling
the function, the array is checked against the signature (Int,
Str), and so if the array doesn't contain of exactly one Int and one
Str in this order, a type error is thrown.
The same mechanism can be used not only for validation, but also for
unpacking, which means extracting some parts of the data structure.
This simply works by using variables in the inner signature:
sub head(*@ [$head, *@]) {
$head;
}
sub tail(*@ [$, *@tail]) {
@tail;
}
say head <a b c >;
say tail <a b c >;
Here the outer parameter is anonymous (the @), though it's
entirely possible to use variables for both the inner and the outer
parameter.
The anonymous parameter can even be omitted, and you can write sub
tail( [$, *@tail] ) directly.
Sub-signatures are not limited to arrays. For working on arbitrary objects,
you surround them with parenthesis instead of brackets, and use named
parameters inside:
multi key-type ($ (Numeric :$key, *%)) { "Number" }
multi key-type ($ (Str :$key, *%)) { "String" }
for (42 => 'a', 'b' => 42) -> $pair {
say key-type $pair;
}
This works because the => constructs a Pair, which has a
key and a value attribute. The named parameter
:$key in the sub-signature extracts the attribute
key.
You can build quite impressive things with this feature, for example red-black tree
balancing based on multi dispatch and signature unpacking. (More verbose explanation of the code.) Most use
cases aren't this impressive, but still it is very useful to have
occasionally. Like for this small
evaluator.
February 13, 2013 05:49
February 09, 2013
This time without any gimmicks (like five word sentences). This will be simple description of changes I do regularly.
New features (this time without building stuff part)
Rakudo Perl
- Constant folded
^..^ (infix) and ^ (prefix) operators.
Niecza Perl
- Added (borrowed from Rakudo Perl) the
Match.gist method.
- Replaced “Unhandle conext character” error with “Unhandle context character” error.
- Added Cool.set method. It can be used to convert argument to set.
- Added Cool.bag. Just like method above, it can be used to convert argument to bag.
February 09, 2013 08:00
February 06, 2013
Yet another ignorable self promotion. http://twitter.com/GlitchMr
(perhaps I should have different news feed for Planet Six…)
February 06, 2013 08:00
February 02, 2013
The changes are so fun. jnthn is working on porting. nqp will be on JVM. And then, Rakudo will be. Java is very, very fast. There are some interesting problems. But nothing impossible to solve. At least, I hope so. I would continue, but well. This is not the topic. This is article about changes. Changes in Perl 6, obviously. If you want, enjoy reading.
Rakudo Star 2013.01 was released. It’s based on Rakudo 2013.01. Those changes aren’t in.
On sidenote, a small note. Five word sentences are interesting. I am not stealing ideas. Well, actually, I am now. I should stop talking now.
New features to build stuff
Rakudo, a Perl 6 distribution
any and all are optimized. To be exact, in statements. The if and while statements. And opposites - unless and until.- Many operators are pure now. That means they are sideeffect-free.
- Site libraries are loaded first.
- Pure subroutines are constant folded.
- REPL supports natively typed variables.
- REPL has better error reporting.
February 02, 2013 08:00
February 01, 2013
After my last post giving a status update on the JVM porting of NQP and the compiler toolchain Rakudo builds upon, hercynium++ left a comment suggesting that I also blog about the design work behind this effort. I liked the idea, and in this post I’ll attempt to describe it a bit. I can’t possibly capture all of the interesting things in a single post, so if this doesn’t cover aspects that are particularly interesting to anybody reading, let me know and I’ll try and find time to write something on them. :-)
It started long ago…
The first commit to the repository where I’m doing the initial porting work to the JVM may have been back in November, but that isn’t where the journey really started. We’ve known for multiple years now that we would want Rakudo and NQP to target backends besides Parrot. In that time, we’ve had to build a lot of technology in order to be able to build Rakudo at all. Some things we’ve had to build more than once because the first time didn’t produce something satisfactory (where satisfactory means “actually meets our needs”, not “is the most awesome thing ever possible”). Software is, fairly often, as much about learning as it is about building. The more complex the domain you’re working in, there more this applies, and the more likely it is that you’ll have to build one to throw away. By now we’ve thrown away a parser engine, an AST, and about 3 implementations of roles. :-)
Of course, there’s the build/buy thing, where buy in open source really means “buy into”, as in use an existing library. We’ve done a bunch of that too, such as libtommath for our big integer support and dyncall for NativeCall support. But the closer something is to the “core domain” – the thing that makes your product distinctive and special – the less able you are to use something off the shelf. Parsing Perl 6 really needs to be done with a Perl 6 grammar, using Longest Token Matching. Its object system really needs something that supports meta-programming, representation polymorphism and gradual typing. Getting BEGIN/eval right and supporting compilation and having the possibility for lexical and anonymous types and packages, which can be dynamically constructed and exported, also left us with something to build (this is the work that led to bounded serialization).
Eventual portability has been a design factor in what we’ve built for quite a while. While the only 6model implementation to have become complete enough to support all of Rakudo’s object needs so far is the one running on Parrot, the initial prototypes of 6model were done on the .Net CLR. This was in no small part to make sure that there was a feasible way to implement it on such a VM. Granted, what I actually discovered was a less than awesome way to build it on the CLR (and what I’m doing on the JVM this time around fits in far better with the JVM’s world view). But it was a design consideration from the start.
When we updated PAST, the previous AST representation, to QAST (Q is just P++ :-)) then once again portability was a concern; the VM specific bits were all placed under a QAST::VM node type. This makes it easy to escape to the underlying VM where needed or where it is most expedient, but it’s both explicit and done in a way that allows specification of what to do on other backends. As part of this work we also build support for the nqp::op abstraction directly into the AST format. The nqp::ops form an opcode set independent of any particular VM. These get mapped as part of turning a QAST tree into code for the target backend (thus meaning there’s no overhead for them in the generated code). They may map directly to the VM’s opcodes, a function or method call in the VM, or do some more complex code generation.
The other important piece of the groundwork for portability is that we implemented Rakudo in a mixture of Perl 6 and NQP, and over time have got NQP to the point where it is also written in NQP (and thus can compile itself). This has been a gradual thing; the earliest NQP editions were written directly in PIR, and with time we’ve migrated those bits to NQP – usually at the same point we were doing other improvements already. For example, pmichaud++ wrote the latest edition of the regex engine, with LTM support, in NQP. PAST, written in PIR, was replaced by QAST, written in NQP. And 6model’s meta-objects were, from the start, expressed in NQP too. It’s pretty neat that NQP’s definition of things so fundamental as classes is actually written in NQP. It means that we don’t have to port classes and roles, just the primitives they are made out of.
So digging into the JVM port itself…
With all of the above mentioned things in place, it was possible to form a fairly concrete roadmap for porting NQP, then Rakudo, over to the JVM. Being comfortable that the result would enable us to get a fully functional Rakudo on the JVM and an idea of how to get there was important. It’s easy to implement a subset, but if it isn’t factored in a way that lets you do the rest afterwards then you’re in bother and it’ll be time for re-work. My hope was that, after some years of learning about things that don’t work and replacing them with things that do, this time much of the re-work could be avoided. A starting point for this was taking a good look at the JVM’s instruction set, as well as considering what JVMs are typically good at doing.
The JVM is a stack machine. This is in contrast to Parrot, which is a register machine. Thankfully, this is mostly a code generation detail rather than being especially deep. As well as the stack, a given method can have local variables (note that everything that contains code on the JVM is called a method, even subroutines, but they call them static methods because it sounds so much more OO :-)). These can hold longer-lived things, so in a sense could be used a bit like Parrot registers. In general, the code generation from QAST uses the stack where possible and falls back to locals where needed. This is because stack usage fits well with what a JVM expects to be doing, and also what its bytecode format expresses most compactly.
Locals have an important restriction: they can only be accessed directly in the scope where they are declared. There is no notion of nested methods at the JVM level. This means that locals are not suitable for implementing lexical variables. Thankfully, there is a well established solution: promote such things onto the heap, keeping them in some kind of invocation record. This is what happens with closures in C# on the CLR, for example. There are a bunch of ways to do this transform, with various trade-offs. I’ve done one that was fairly fast to implement, but also enables lookup by array indexing rather than needing a named (hash) lookup in the vast majority of cases. As well as an array index being algorithmically cheaper than a hash lookup, the JVM supports array indexing natively in its opcode set, but not hash lookups.
Talking of allocating things on the heap brings us nicely to think about objects. JVMs are very good at fast allocation and collection of objects, because they have to be; there is no stack allocation in Java of anything non-trivial. Of course, that doesn’t mean the VM can’t do escape analysis and stack allocate under the hood. That the VM is really good at object allocation and GC means we don’t need to worry too much about lexicals leading to invocation records on the heap; there’s plenty of performant uses of this approach in the wild. Furthermore, most records will be very short lived, nicely meeting the generational hypothesis (which is that most objects are either short lived or long lived, and so we can optimize separately for each through doing generational garbage collection).
While invocation records are relatively internal, of course NQP and Perl 6 involve lots of user-visible objects. From the things you think about as objects (and call “new” on) to things like scalar containers, strings, boxed integers and so forth, both NQP and Perl 6 lead to plenty of allocations. While some things are quite predictably shaped, most come from user class definitions. Ideally, we’d like it if a Perl 6 class definition like:
class Point {
has $!surface;
has num $!x;
has num $!y;
}
Was to use memory similarly to if you wrote something in Java like:
class Point {
private Object surface;
private double x;
private double y;
}
At the same time, we know that the JVM’s idea of type is some way off the Perl 6 notion of type, so we can’t simply turn Perl 6 classes into JVM classes. Thankfully, 6model has from the start been designed around the idea of representation polymorphism. Really, this is just a separation of concerns: we decouple the details of memory representation and access from the notion of being a type and dispatch. The former is handled by a representation, and the latter two by a meta-object. One early but important observation I made when designing 6model is that the representation will always couple closely to the underlying runtime (and thus would need to be implemented for each runtime we wanted to run on), whereas the other bits can be expressed in a higher level way, with the common cases made efficient by caching. Thus there’s no reason to re-implement classes and roles per VM, but there is a need to provide a different, VM-specific way to do P6opaque (the default representation for NQP and Perl 6 objects).
The C implementation of P6opaque on Parrot works by calculating a memory layout – essentially, figuring out a struct “on the fly”. What’s the JVM equivalent of that? Well, that’s just creating a JVM class on the fly and loading it. Is the JVM capable of doing that? Sure, it’s easily dynamic enough. Furthermore, once we’ve done that little bit of bytecode generation, it’s a perfectly ordinary JVM class. This means that the JIT compiler knows what to do with it. Does doing any of this require changes to the meta-objects for classes in NQP and Rakudo? No, because these details are all encapsulated in the representation. Things like these are good signs for a design; it tends to show that responsibilities are correctly identified and segregated.
So, how’s the cross-compiler going?
Things are going nicely. Having got much of the way there with the NQP MOP, I turned to ModuleLoader and started to get together a basic setting (the setting being the place where built-ins are defined). With those in place, work has moved on to trying to pass the NQP test suite.
The build process cross-compiles the MOP, module loader and setting. To run the test suite, each test is taken and cross-compiled against those, then the result of compiling it is run on the JVM. The fact we invoke NQP, then invoke the JVM twice in order to run each test gives quite a bit of fixed overhead per test; once we have NQP itself (that is, the compiler) cross-compiled and self-hosting on the JVM it’ll be down to a single invocation.
The NQP test suite for the NQP language itself consists of 65 test files. 3 of them are specific to Parrot, so there’s 62 that are interesting to make run. As of today, we pass 46 of those test files in full. While some of those passing tests exercise relatively simple things (literals, operators, variables, conditionals, loops, closures), others exercise more advanced features (classes, roles, basic MOP functionality, runtime mixins and so forth). Of the 16 test files that remain, 9 of them depend on regexes or grammars. Getting those to run will be the focus of the next major chunk of work: porting the regex compiler and getting the NFA, Cursor and Match classes to cross-compile (which will involve some portability improvements). The other 7 relate to non-trivial, but smaller-than-grammars features (for example, 2 are about multiple dispatch, which I’m working on porting at the moment).
It was only three weeks ago when I wrote that the JVM port did not even manage “hello world” yet, and that I had little more to show than something that could turn a range of QAST trees into JVM bytecode. Three weeks later and we’re running around 75% of the NQP test files, and timotimo++ was even able to feed an almost unmodified Levenstein distance implementation written in NQP to the cross-compiler and have it run on the JVM.
So, mad amounts of coding have taken place? Well, only sorta…I’ve taught two three-day classes for $dayjob in the last couple of weeks also. :-) Mostly, progress has been fast now because the foundations it is building upon have proved fairly solid. For the backend, this is in no small part down to having grown a test suite for the QAST to JVM phase of the work as it progressed. The fact we could happily couple this new backend to the existing NQP parser is thanks to the compiler being structured as a pipeline of stages, each one strongly isolated from the others, just passing a data structure between them. In my teaching work, I often encourage automated testing and talk a lot about the importance of enforcing carefully chosen, well-defined, information-centric boundaries between components. It’s pleasing to see these things paying off well in my Perl 6 work also. :-)
February 01, 2013 23:49
January 31, 2013
(Using exclusively five word sentences.)
Perl Mongers needed speakers quickly. "Very Late Call for Papers". "Why so late", you ask. Perl dev room was denied. Another community got the room. Perl only got a booth. The other community backed out. Perl then got the room. Therefore, talks were requested urgently. Only about one week notice. The announcement is recorded here.
I missed that blog post. But I got an email. Wendy wrote to some people. I was one of them. Talk about very short notice. Eight days before the talk! That must be a record. Nothing to be done, though. The invitation was nicely worded. I considered whether to go. Finally I decided I would.
My talk concerns Perl 6. I have given it before. It was in Bristol, England. You were likely not there. That time, jnthn helped me. Now I will talk alone. I must give it quickly. I only have 20 minutes. That is not a lot. I rather like challenges, though. Looking forward to it all.
Will you come to FOSDEM? I certainly hope you will. If you do, stop by. I will give my talk. "Where is my flying car?" A reference to the future. In the future, cars fly. Also, Perl 6 is everywhere. Especially in the flying cars. It will be totally awesome. My talk is about that. Or sorta kinda about that. It is about Perl 6. Why is it not released? What makes me keep hoping? What has been implemented already? That is what it covers.
Looking forward to the weekend.
January 31, 2013 23:22
January 30, 2013
On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the January 2013 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the January 2013 release is available from the download page. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.
In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2013.01 [0] of the Rakudo Perl 6 compiler [1], version 4.10.0 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community.
Some of the new features added to this release include:
- Sink context (what some other languages call void context) is now enforced correctly. This means that for-loops are now lazy by default. It fixes the bug where a map in sink context would not execute, and also means that a Failure returned to sink context will be properly thrown.
- ‘require’ now works with indirect module names
- Restored socket read semantics to returning the requested number of bytes
- $obj.Some::Role::meth() now passes the correct $obj
- try/CATCH now returns Nil when the CATCH is triggered, rather than the exception; this brings it in line with try without a CATCH
- whatever-star cases of splice now implemented
- can now import multis with the same name from different modules, provided all dispatchers are onlystar
This release also contains a range of bug fixes, improvements to error reporting
and better failure modes.
The following features have been deprecated or modified from previous
releases due to changes in the Perl 6 specification, and are being removed
or changed as follows:
- postcircumfix:<[ ]> and postcircumfix:<{ }> will become multi-subs rather than multi-methods. Both at_pos and at_key will remain methods.
- Unary hyper ops currently descend into nested arrays and hashes. This will change to make them equivalent to a one-level map.
- The Str.ucfirst builtin is deprecated; it will be replaced by Str.tc.
- Leading whitespace in rules and under :sigspace will no longer be
converted to <.ws> . For existing regexes that expect this conversion, add a <?> in front of leading whitespace to make it meta again.
- The ?-quantifier on captures in regexes currently binds the capture slot to a List containing either zero or one Match objects; i.e., it is equivalent to “** 0..1″. In the future, the ?-quantifier will bind the slot directly to a captured Match or to Nil. Existing code can manage the transition by changing existing ?-quantifiers to use “** 0..1″, which will continue to return a List of matches.
There are some key features of Perl 6 that Rakudo Star does not
yet handle appropriately, although they will appear in upcoming
releases. Some of the not-quite-there features include:
- advanced macros
- threads and concurrency
- Unicode strings at levels other than codepoints
- interactive readline that understands Unicode
- non-blocking I/O
- much of Synopsis 9
There is an online resource at http://perl6.org/compilers/features
that lists the known implemented and missing features of Rakudo
and other Perl 6 implementations.
In many places we’ve tried to make Rakudo smart enough to inform the
programmer that a given feature isn’t implemented, but there are
many that we’ve missed. Bug reports about missing and broken
features are welcomed at .
See http://perl6.org/ for links to much more information about
Perl 6, including documentation, example code, tutorials, reference
materials, specification documents, and other supporting resources.
A draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf
in the release tarball.
The development team thanks all of the contributors and sponsors
for making Rakudo Star possible. If you would like to contribute,
see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org
mailing list, or join us on IRC #perl6 on freenode.
[0] https://github.com/rakudo/rakudo/blob/nom/docs/announce/2013.01
[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/
January 30, 2013 18:51
January 26, 2013
Today, the main features is .delta method in DateTime and Date. And as usually, bug fixes that make language do what you mean.
New features
Perl 6 specification
- Added
DateTime.delta and Date.delta to the specification.
Rakudo Perl
s/// resets $/ on failure.s/// returns False on failure.- Regular expressions allow
$() syntax for arbitrary expressions.
- Socket functions work better with non-ASCII characters.
- Errors about function only taking positional arguments now specify name of function that was called.
DateTime.truncated-to now takes TimeUnit enum as argument.- Implemented
DateTime.delta and Date.delta.
- Names are suggested for
--> Type (subroutine return type).
/< hello >/ now works correctly (matching hello), just like /< hello world >/ already did.
January 26, 2013 08:00
January 20, 2013
Rakudo lately got an interesting change. Consider following buggy code (for array binary search), written in functional style (I’m using recursion). The bug is that it doesn’t work because of two different identifiers binary-search and binary_search.
Trying to run it shows compile time error message. The compile part is important. If you would remove binary-search call at end, it still would report an error, unlike Python.
===SORRY!===
Undeclared routine:
binary_search used at lines 2, 8, 9. Did you mean '&binary-search'?
Now you know what’s wrong and you can easily fix it. Unlike let’s say, Jekyll that I use for my blog. Not only it doesn’t work on my PC for some reason, but also refused to highlight my code - it gave some sort of XML error (it simply said “REXML could not parse this XML/HTML”). I gave up and put it on Gist. And the first attempt was failed, because embed code is <script> and Jekyll self-closed the tag (XML-style, but this is HTML). This is annoying.
January 20, 2013 08:00
January 19, 2013
"I've found this year the tasks were harder, but I've also spend more time thinking about the problems and less worrying about bugs in the interpreter... which is definitely good."
— a contestant
"This year's problems were a lot more approachable to me and motivated me to learn more perl6."
— another contestant
...aaaand we're done. The five weeks are up.
I'm now all caught up processing submissions sent to me. So let's summarize, as usual:
- 32 people signed up as contestants this year.
- 8 contestants submitted solutions.
- 24 submissions passed a
base-test run and were accepted.
These figures during p6cc2010 and p6cc2011 were (18, 5, 26) and (35, 6, 27), respectively. So all in all, it seems that we've entered a steady state both in terms of contestants and submissions. I'm grateful to get so much interesting Perl 6 code to read through and review.
Many people told me this year that they had noble plans to send in lots of solutions, but in the end didn't find the time for it. I'm assuming many contestants I haven't heard from have similar stories. That's fine; the contest is designed to encourage you to do the tasks, but not to force you to. If the contest made you take a look at the tasks, or at Perl 6, then in some sense that's a win, too.
My plan is, just like previous years, to go through the five tasks in order, publish all the reviews and an appropriately thoughtful blog post that summarizes the tasks and its solutions. With luck, we'll all learn something together.
It takes time to do these reviews. Sometimes a lot of time. That said, I hope to get through these submissions quickly. Expect something like a post each week. That's the goal.
After which I'll sum up and select a winner, also in a blog post. Exciting!
So, expect the next blog post to be about the first task: "Tell knights from knaves based on what they say." Onwards!
January 19, 2013 17:24
It’s another Perl 6 changes article (at this point I wonder why I still do that). But as long the changes are impressive (and the cake is the lie), I guess mentioning changes is worth it.
Rakudo 2013.01 was tagged in the Git repository. It wasn’t yet released, but it’s really close to the releease (and frozen).
New features
Rakudo Perl (in Rakudo 2013.01)
** 0 in regexpes works correctly matching nothing.
Rakudo Perl (post Rakudo 2013.01)
- Compiler hints invalid variable names. For example, typing
$variabel when you already have $variable declared hints that variable.
Niecza Perl
- More meaningful error message is thrown when
printf tries to consume more arguments than available.
January 19, 2013 08:00
January 18, 2013
Things have been moving along quite rapidly on the JVM backend since my last post. Sadly, I’m too sick to hack on anything much this evening (hopefully, this turns out to be a very temporary affliction…) but I can at least just about write English, so I figured I’d provide a little update. :-)
Last time I blogged here, I was able to compile various QAST trees down to JVM bytecode and had a growing test suite for this. My hope was that, by some inductive handwaving, being able to compile a bunch of QAST nodes and operations correctly would mean that programs made up of a whole range of them would also compile correctly. In the last week or so, that has come to pass.
Having reached the point of having coverage of quite a lot of QAST, I decided to look into getting an NQP frontend plugged into my QAST to JVM backend. In the process, I found that NQP lacked the odd VM abstraction here and there in the common prelude that it includes with every QAST tree it produces. Thankfully, this was easily rectified. Even better, I got rid of a couple of old hacks that were no longer required. With those things out of the way, I found that this common prelude depended on a couple of operations that I’d not got around to implementing in the JVM backend. These were also simple to add. And…here endeth the integration story. Yup, that was it: I now had a fledgling NQP cross-compiler. An NQP compiler running on Parrot, but producing output for the JVM.
This result is rather exciting, because…
- It’s using exactly the same parse and action stages as when we’re targeting Parrot. No hacks, no fork. The QAST tree we get from the NQP source code that goes in is exactly the one we get when targeting Parrot. Everything that happens differently happens is beyond that stage, in the backend. This is an extremely positive sign, architecturally.
- With a couple of small additions to handle the prelude, I was immediately able to cross-compile simple NQP programs and run them on the JVM. There’s no setting or MOP yet, but the basics (variables, loops, subroutines with parameters, even closures) Just Worked.
- The program I wrote to glue the JVM backend work and the existing NQP frontend together was about 30 lines of NQP code.
- This whole integration process was about an afternoon’s worth of work.
Since I got that working, my focus has been on getting nqp-mo (the NQP meta-objects) to cross-compile. This is where classes and roles are implemented, and thus is a prerequisite for cross-compiling the NQP setting, which is in turn a prerequisite for being able to start cross-compiling and running the NQP test suite. The NQP MOP is about 1500 lines of NQP code, and at this point I’ve got about 1400 of them to cross-compile. So I’m almost there with it? Well, not quite. Turns out that the next thing I need to port is the bounded serialization stuff. That’s a rather hairy chunk of work.
Anyway, things are moving along nicely. The immediate roadmap is to get the bounded serialization to the point where it’s enough for the NQP MOP, then move on to getting a minimal setting cross compiling. Beyond that, it’ll be working through the test suite, porting the regex compilation and seeing what else is needed to cross-compile the rest of NQP.
January 18, 2013 00:44
January 12, 2013
I’m going to show another Perl 6 changes article. I’m so lazy, that I really don’t know what to put here, so I guess I’ll now show the list of new features.
New features
Perl 6 specification
--> (return type) acts more like a normal subroutine parameter.- Added
squish method, working like uniq utility in UNIX - it only removes elements if it is duplicate of previous element.
- Adjusted precedence of set operators.
STD.pm6
- Adjusted precedence of set operators.
Rakudo Perl
- Implemented
require ::($module) 'something' syntax.
$socket.read($bytes) now returns $bytes bytes, if possible. Previously it only read only 256 bytes.$object.My::Role::method() works correctly and uses correct self.try with CATCH returns Nil on exception, like it did without explicit CATCH.- Failures in sink context won’t throw if they were handled.
splice supports whatever star (*).- Adjusted precedence of set operators.
Junction.Str now gives string, instead of junction with every element converted to the string.
Niecza Perl
- Adjusted precedence of set operators.
January 12, 2013 08:00
January 10, 2013
Seems it’s high time for some news here. It’s not that I didn’t do any blogging about Perl 6 in December; it’s just that all of those posts were over on the Perl 6 advent calendar. Anyway, now it’s a new year, and I’m digging back into things after an enjoyable Christmas and New Year’s break in the UK with family and friends. Here’s a bunch of things that already happened but I didn’t get around to talking about here yet, and some things that will be coming up.
Better Parse Errors In 2012.12
Ever had Rakudo tell you there’s a problem on line 1, when it’s really on line 50? Or wished that even in the common case where it gets the line right, it would tell you exactly where on the line things went wrong? Or how about the time it told you “Confused” because you got a closing paren too many?
Many of my contributions to the 2012.12 Rakudo release centered around improving its reporting of parse errors. STD, the standard Perl 6 grammar, has had much better error reporting than Rakudo for a while. Therefore, I spent a bunch of time aligning our error reporting more closely with what STD does. Some of this is cosmetic: you get the colored output and the indication of the parse location. But while these cosmetic changes will be the most immediately visible thing, the changes go far deeper. Of note, a high water mark is kept so we can be a lot more accurate in reporting where things came unstuck, and we track what was expected so we can produce better errors. Just doing the cosmetic stuff without being able to give it a better location to report wouldn’t have helped so much. :-)
One other change is that we don’t bail out on the first thing that’s wrong when it’s possible to survive and continue parsing. When this is possible, up to 10 errors will be reported (since that’s typically a screen worth). Of course, some things just hose the parse and we can’t continue in any sensible way.
Hopefully, these improvements will make using Rakudo feel a lot nicer. Already on channel, I can feel the feedback we’re giving about parse errors when people use the evalbot is often a lot more pleasant and informative. Of course, there’ll be more improvements in the future too, but this is a big step forward.
Faster Auto-Threading
The junction auto-threader could sometimes be insanely slow. As in, ridiculously so. After hearing a bunch of reports about this, I decided to dig in and work out why. A rewrite later, the little benchmark I was using with it ran almost 30 times faster. Not so bad… :-) This change also made it into the 2012.12 release.
JVM Backend Preparations Underway
I’ve talked plenty about plans for NQP and Rakudo to run on things besides Parrot for a while now. Over the last year or two, we’ve laid a lot of the groundwork for this. What’s been especially pleasing is that it’s also made Rakudo a better quality Perl 6 implementation on Parrot, thanks to the many architectural improvements. Of note, in many places we’ve closed semantic gaps between what Perl 6 wants and the primitives we were building it out of; the new QAST (Q Abstract Syntax Tree) is a great example.
Anyway, with NQP now being written pretty much entirely in NQP, and many of the right abstractions in place, it felt like time to start slowly picking away at getting 6model ported to the JVM and work on turning QAST trees into Java bytecode. I quietly started on this in November, and mentioned the existence of the work on #perl6 in December. I was delighted to see Coke++ jump in and start working through the Low Hanging Fruit – a file where I’m putting tasks that should be relatively easy to pick off. I actually had to re-fill it, after the last round were depleted. ;-) By now, quite a few bits of QAST are supported and the 6model on JVM implementation is coming along nicely. Yes, this means it’s already capable of doing basic meta-programming stuff.
Note that this work isn’t at the stage where it’s of use for anything yet. You can’t even write a say(“hello world”) in NQP and have it run on the JVM yet, since all the work so far is just about turning QAST trees into JVM bytecode and building the runtime support it needs. That may seem like a curious way to work, but once you do enough compiler stuff you find yourself thinking quite naturally in trees. It meant I didn’t have to worry about creating some stripped-down NQP that could emit super-simple trees to be able to test really simple things. After all, the goal is to run NQP itself on the JVM, and then Rakudo, and only then will things be interesting to the everyday user.
To address a couple of immediate concerns that some may have…
- No, this is not a case of “stop running on Parrot, start running on JVM”. It’s adding an additional backend, much like pmurias++ has been working on adding a JavaScript backend for NQP. Of course, I expect resource allocation in the future to be driven by which backends users desire most. For some, the JVM is “that evil Oracle thing” and they don’t want to touch it. For others, the JVM is “the only thing our company will deploy on”. Thus I expect this work to matter more to some people than others. That’s fine.
- No, targeting multiple backends doesn’t mean performance-sucking abstractions everywhere. It’s a pretty typical thing for a compiler to do. As usual, it’s about picking the right abstractions. The debugger was implemented as an additional Rakudo frontend without a single addition to Rakudo or NQP or anything anywhere in the stack. That was possible because things were designed well. I’m sure the process of getting things running on the JVM will flag up a few places where things aren’t designed as well as they need to be, but already I’m seeing a lot of places where things are mapping over very nicely indeed.
- No, this doesn’t mean that all other Rakudo development will grind to a halt. I’ve been working on the JVM backend stuff in November and December; both months saw a huge amount of Rakudo progress too. Things will go on that way.
Type System Improvements
Rakudo does a lot of things well when it comes to supporting the various kinds of types Perl 6 offers, but there are some weak areas. Here are some of the things I plan to focus on:
- Getting the definedness constraint stuff working better (the :D and :U annotations). At the moment, they’re supported as a special case in multiple dispatch and the signature binder. In reality, they’re meant to be first class and work everywhere. You may not think you care about these much. Actually, you probably at least indirectly do, because once the optimizer is able to analyze them, it’ll be able to do a bunch more inlining of things than it can today. :-)
- Getting coercion types in place. Again, this is turning the special-cased “as” syntax into the much more general coercion type syntax (for example, Int(Str) is the type that accepts a Str and coerces it into an Int).
- Getting native types much better supported. At the moment, you can use them but…there are pitfalls. Having them available has been a huge win for us in the CORE setting, where we’ve used them in the built-ins. But they’re still a bit “handle with great care”. I want to change that.
- Implementing compact arrays.
- Improving our parametric type and type variable support. Many things work, but there are some fragile spots and some bugs that want some attention.
I intend to have some of these things in the January release, and a bunch more in the February one. We’ll see how I get along. :-)
January 10, 2013 01:59
January 05, 2013
I have finally updated the number before ‘W’. Well, we have the new year - 2013. And because of that I would like to say ‘Happy new year!’.
This is yet another article in “Perl 6 changes”. I lately don’t put articles other than those, but I don’t have many ideas for those, really.
By the way, I find it interesting that no new posts on Planet Six appeared between my previous Perl 6 changes article and current Perl 6 changes article.
Incompatible changes
Rakudo Perl
- “SAFE” setting is now removed. You’re supposed to use “RESTRICTED” setting now.
New features
Rakudo Perl
- Using non-associative operators when association is needed throws typed error message.
- Compile time optimizations for
(|) signature (accepting anything), It’s worth handling because it’s commonly used for protos.
- Fixed typos in
X::Import::Redeclaration error message.
- Removed old
perl6doc documentation.
DateTime.Date now works correctly..pick now randomizes better on large ranges.
Niecza Perl
infix:<->(Instant, Instant) now supported.- Bitwise operations support big integers.
January 05, 2013 08:00
December 29, 2012
This is last “Perl 6 changes” update in this year. Enjoy.
New features
Perl 6 specification
uniq method/subroutine, existing for long time in implementations is now mentioned in specification (with === semantics).
Rakudo Perl
- Use of suffix regex modifiers (P5-ism) in
s///, like s////g is now detected.
December 29, 2012 08:00
December 27, 2012
On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the December 2012 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the December 2012 release is available from the download section. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.
In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2012.12 of the Rakudo Perl 6 compiler, version 4.10.0 of the Parrot Virtual Machine, and various modules, documentation, and other resources collected from the Perl 6 community.
Some of the new features added to this release include:
- Parse errors are much improved, and follow STD, the standard parser,
much more closely; they are more accurate and more information is given
- Rakudo now keeps parsing after some less serious errors
- Better errors for various parse failures
- The junction autothreader is now an order of magnitude faster
- Texas (ASCII) versions of the Set and Bag operators implemented
- Nested Pairs now give correct .perl output
-
{ a => $_ } now correctly considered a block, not a hash as
before
This release also contains a range of performance improvements, bug fixes, improvements to error reporting and better failure modes.
The following features have been deprecated or modified from previous releases due to changes in the Perl 6 specification, and are being removed or changed as follows:
- ‘for’-loops will become lazy, and are only evaluated eagerly in eager or sink (void) context. This means that if a for-loop is the last statement in a routine, it will usually run after the routine has returned, so it cannot call return() anymore.
- Unary hyper ops currently descend into nested arrays and hashes. This will change to make them equivalent to a one-level map.
- The Str.ucfirst builtin is deprecated; it will be replaced by
Str.tc.
- Leading whitespace in rules and under :sigspace will no longer be converted to <.ws> . For existing regexes that expect this conversion, add a <?> in front of leading whitespace to make it meta again.
- The ?-quantifier on captures in regexes currently binds the capture slot to a List containing either zero or one Match objects; i.e., it is equivalent to “** 0..1″. In the future, the ?-quantifier will bind the slot directly to a captured Match or to Nil. Existing code can manage the transition by changing existing ?-quantifiers to use “** 0..1″, which will continue to return a List of matches.
There are some key features of Perl 6 that Rakudo Star does not
yet handle appropriately, although they will appear in upcoming
releases. Some of the not-quite-there features include:
- advanced macros
- threads and concurrency
- Unicode strings at levels other than codepoints
- interactive readline that understands Unicode
- non-blocking I/O
- much of Synopsis 9
There is an online resource that lists the known implemented and missing features of Rakudo and other Perl 6 implementations.
In many places we’ve tried to make Rakudo smart enough to inform the programmer that a given feature isn’t implemented, but there are many that we’ve missed. Bug reports about missing and broken features are welcomed at rakudobug@perl.org.
See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. A draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf in the release tarball.
The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.
December 27, 2012 19:53
December 24, 2012
Recently I was unpacking some boxes of books and came across a book entitled "BASIC Computer Programs for the Home" by Charles D. Sternberg. Apparently my father had purchased this book in the early 1980s and had given it to me. In any case, my name was scrawled in the front cover in the manner an adolescent me would have done.
Mostly this book is filled with simple BASIC programs that manage simple databases of various things: recipes, household budget, address book, music collections, book collections, school grades, etc. But the program that caught my eye and made me think of the Perl 6 Advent Calendar was one for printing a calendar starting at a particular month.
Now, the version in this book is a little simple in that it asks for the starting month, year, the day of the week that the first month starts on, and how many months to print. I wanted something a little more like the Unix utility cal(1) program. Luckily, Perl 6 has date handling classes as part of the specification and both major implemenations, Rakudo and Niecza, have actual implementations of these which should make creating the calendar quite easy.
For reference, the output of the Unix cal(1) utility looks like this:
December 2012
Su Mo Tu We Th Fr Sa
1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31
It also has some options to change the output in various ways, but I just want to focus on reproducing the above basic output.
I'll need a list of month names and weekday abbreviations:
constant @months = <January February March April May June July
August September October November December>;
constant @days = <Su Mo Tu We Th Fr Sa>;
And it looks like the month and year are centered above the days of the week. Generating a calendar for May shows this to be the case, so I'll need a routine that centers text:
sub center(Str $text, Int $width) {
my $prefix = ' ' x ($width - $text.chars) div 2;
my $suffix = ' ' x $width - $text.chars - $prefix.chars;
return $prefix ~ $text ~ $suffix;
}
Now, the mainline code needs two things: a month and a year. From this it should be able to generate an appropriate calendar. But, we should have a reasonable default for these values I think. Today's month and year seem reasonable to me:
sub MAIN(:$year = Date.today.year, :$month = Date.today.month) {
but if it's not today's month and year, then it's some arbitrary month and year we need info about. To do this we construct a new Date object from the month and year given.
my $dt = Date.new(:year($year), :month($month), :day(1) );
Looking at the calendar generated for December, it seems like we may actually output up to 6 rows of numbers since the month can start and end on a partial week. In order to implement this, I think I'll need some "slots" for each day. Each slot will either be empty or will contain the day of the month. The number of empty slots at the beginning of the month correspond to the day of the week that the first of the month occurs on. If the first is on Sunday, there will be 0 empty slots, if the first is on a Monday there will be 1 empty slot, if the first is on a Tuesday, there will be 2 empty slots, etc. This is remarkably similar to the number we get when we interrogate a Date object for the day of the week. The only wrinkle is that it returns 7 for Sunday when we actually need a 0. That's easily remedied with a modulus operator however:
my $dt = Date.new(:year($year), :month($month), :day(1) );
my $ss = $dt.day-of-week % 7;
my @slots = ''.fmt("%2s") xx $ss;
That gives us the empty slots at the beginning, but what about the ones that actually contain the days of the month? Easy enough, we'll just generate a number for each day of the month using the Date object we created earlier.
my $days-in-month = $dt.days-in-month;
for $ss ..^ $ss + $days-in-month {
@slots[$_] = $dt.day.fmt("%2d");
$dt++
}
Now we've got an array with appropriate values in the appropriate positions, all that's left is to actually output the calendar. Using the header line for our weekdays as a metric for the width of the calendar, and the routine we created for centering text, we can output the header portion of the calendar:
my $weekdays = @days.fmt("%2s").join: " ";
say center(@months[$month-1] ~ " " ~ $year, $weekdays.chars);
say $weekdays;
Then we iterate over each slot and output the appropriate values. If we've reached the end of the week or the end of the month, we output a newline:
for @slots.kv -> $k, $v {
print "$v ";
print "\n" if ($k+1) %% 7 or $v == $days-in-month;
}
Putting it all together, here is the final program:
#!/usr/bin/env perl6
constant @months = <January February March April May June July
August September October November December>;
constant @days = <Su Mo Tu We Th Fr Sa>;
sub center(Str $text, Int $width) {
my $prefix = ' ' x ($width - $text.chars) div 2;
my $suffix = ' ' x $width - $text.chars - $prefix.chars;
return $prefix ~ $text ~ $suffix;
}
sub MAIN(:$year = Date.today.year, :$month = Date.today.month) {
my $dt = Date.new(:year($year), :month($month), :day(1) );
my $ss = $dt.day-of-week % 7;
my @slots = ''.fmt("%2s") xx $ss;
my $days-in-month = $dt.days-in-month;
for $ss ..^ $ss + $days-in-month {
@slots[$_] = $dt.day.fmt("%2d");
$dt++
}
my $weekdays = @days.fmt("%2s").join: " ";
say center(@months[$month-1] ~ " " ~ $year, $weekdays.chars);
say $weekdays;
for @slots.kv -> $k, $v {
print "$v ";
print "\n" if ($k+1) %% 7 or $v == $days-in-month;
}
}
Normally, cal(1) will highlight today's date on the calendar. That's a feature I left out of my calendar implementation but it could easily be added with Term::ANSIColor. Also, there's a little bit of coupling between the data being generated in the slots and the output processing (the slots are all formatted to be 2 characters wide in anticipation of the output). There are some other improvements that could be done, but for a first cut at a calendar in Perl 6, I'm happy. :-)
December 24, 2012 00:00
December 23, 2012
Syntactic macros. The Lisp gods of yore provided humanity with this invention, essentially making Lisp a programmable programming language. Lisp adherents often look at the rest of the programming world with pity, seeing them fighting to invent wheels that were wrought and polished back in the sixties when giants walked the Earth and people wrote code in all-caps.
And the Lisp adherents see that the rest of us haven’t even gotten to the best part yet, the part with syntactic macros. We’re starting to get the hang of automatic memory management, continuations, and useful first-class functions. But macros are still absent from this picture.
In part, this is because in order to have proper syntactic macros, you basically have to look like Lisp. You know, with the parentheses and all. Lisp ends up having almost no syntax at all, making every program a very close representation of a syntax tree. Which really helps when you have macros starting to manipulate those same trees. Other languages, not really wanting to look like Lisp, find it difficult-to-impossible to pull off the same trick.
The Perl languages love the difficult-to-impossible. Perl programmers publish half a dozen difficult-to-impossible solutions to CPAN before breakfast. And, because Perl 6 is awesome and syntactic macros are awesome, Perl 6 has syntactic macros.
It is known, Khaleesi.
What are macros?
For reasons even I don’t fully understand, I’ve put myself in charge of implementing syntactic macros in Rakudo. Implementing macros means understanding them. Understanding them means my brain melts regularly. Unless it fries. It’s about 50-50.
I have this habit where I come into the #perl6 channel, and exclaiming “macros are just X!” for various values of X. Here are some samples:
- Macros are just syntax tree manipulators.
- Macros are just “little compilers”.
- Macros are just a kind of templates.
- Macros are just routines that do code substitution.
- Macros allow you to safely hand values back and forth between the compile-time world and the runtime world.
But the definition that I finally found that I like best of all comes from scalamacros.org:
Macros are functions that are called by the compiler during compilation. Within these functions the programmer has access to compiler APIs. For example, it is possible to generate, analyze and typecheck code.
While we only cover the “generate” part of it yet in Perl 6, there’s every expectation we’ll be getting to the “analyze and typecheck” parts as well.
Some examples, please?
Coming right up.
macro checkpoint {
my $i = ++(state $n);
quasi { say "CHECKPOINT $i"; }
}
checkpoint;
for ^5 { checkpoint; }
checkpoint;
The quasi block is Perl 6′s way of saying “a piece of code, coming right up!”. You just put your code in the quasi block, and return it from the macro routine.
This code inserts “checkpoints” in our code, like little debugging messages. There’s only three checkpoints in the code, so the output we’ll get looks like this:
CHECKPOINT 1
CHECKPOINT 2
CHECKPOINT 2
CHECKPOINT 2
CHECKPOINT 2
CHECKPOINT 2
CHECKPOINT 3
Note that the “code insertion” happens at compile time. That’s why we get five copies of the CHECKPOINT 2 line, because it’s the same checkpoint running five times. If we had had a subroutine instead:
sub checkpoint {
my $i = ++(state $n);
say "CHECKPOINT $i";
}
Then the program would print 7 distinct checkpoints.
CHECKPOINT 1
CHECKPOINT 2
CHECKPOINT 3
CHECKPOINT 4
CHECKPOINT 5
CHECKPOINT 6
CHECKPOINT 7
As a more practical example, let’s say you have logging output in your program, but you want to be able to switch it off completely. The problem with an ordinary logging subroutine is that with something like:
LOG "The answer is { time-consuming-computation() }";
The time-consuming-computation() will run and take a lot of time even if LOG subsequently finds that logging was turned off. (That’s just how argument evaluation works in a non-lazy language.)
A macro fixes this:
constant LOGGING = True;
macro LOG($message) {
if LOGGING {
quasi { say {{{$message}}} };
}
}
Here we see a new feature: the {{{ }}} triple-block. (Syntax is likely to change in the near future, see below.) It’s our way to mix template code in the quasi block with code coming in from other places. Doing say $message; would have been wrong, because $message is a syntax tree of the message to be logged. We need to inject that syntax tree right into the quasi, and we do that with a triple-block.
The macro conditionally generates logging code in your program. If the constant LOGGING is switched on, the appropriate logging code will replace each LOG macro invocation. If LOGGING is off, each macro invocation will be replaced by literally nothing.
Experience shows that running no code at all is very efficient.
What are syntactic macros?
A lot of things are called “macros” in this world. In programming languages, there are two big categories:
- Textual macros. They substitute code on the level of the source code text. C’s macros, or Perl 5′s source filters, are examples.
- Syntactic macros. They substitute code on the level of the source code syntax tree. Lisp macros are an example.
Textual macros are very powerful, but they represent the kind of power that is just as likely to shoot half your leg off as it is to get you to your destination. Using them requires great care, of the same kind needed for a janitor gig at Jurassic Park.
The problem is that textual macros don’t compose all that well. Bring in more than one of them to work on the same bit of source code, and… all bets are off. This puts severe limits on modularity. Textual macros, being what they are, leak internal details all over the place. This is the big lesson from Perl 5′s source filters, as far as I understand.
Syntactic macros compose wonderfully. The compiler is already a pipeline handing off syntax trees between various processing steps, and syntactic macros are simply more such steps. It’s as if you and the compiler were two children, with the compiler going “Hey, you want to play in my sandbox? Jump right in. Here’s a shovel. We’ve got work to do.” A macro is a shovel.
And syntactic macros allow us to be hygienic, meaning that code in the macro and code outside of the macro don’t step on each other’s toes. In practice, this is done by carefully keeping track of the macros context and the mainline’s context, and making sure wires don’t cross. This is necessary for safe and large-scale composition. Textual macros don’t give us this option at all.
Future
Both of the examples in this post work already in Rakudo. But it might also be useful to know where we’re heading with macros in the next year or so. The list is in the approximate order I expect to tackle things.
- Un-hygiene. While hygienic macros are the sane and preferable default, sometimes you want to step on the toes of the mainline code. There should be an opt-out, and escape hatch. This is next up.
- Introspection. In order to analyze and typecheck code, not just generate it, we need to be able to take syntax trees coming in as macro arguments, and look inside of them. There are no tools for that yet, and there’s no spec to guide us here. But I’m fairly sure people will want this. The trick is to come up with something that doesn’t tie us down to one compiler’s internal syntax-tree format. Both for the sake of compiler interoperability and future compatibility.
- Deferred declarations. The sandbox analogy isn’t so frivolous, really. If you declare a class inside a
quasi block, that declaration is limited (“sandboxed”) to within that quasi block. Then, when the code is injected somewhere in the mainline because of a macro invocation, it should actually run. Fortunately, as it happens, the Rakudo internals are factored in such a way that this will be fairly straightforward to implement.
- Better syntax. The triple-block syntax is probably going away in favor of something better. The problem isn’t the syntax so much as the fact that it currently only works for terms. We want it to work for basically all syntactic categories. A solid proposal for this is yet to materialize, though.
With each of these steps, I expect us to find new and fruitful uses of macros. Knowing my fellow Perl 6 developers, we’ll probably find some uses that will shock and disgust us all, too.
In conclusion
Perl 6 is awesome because it puts you, the programmer, in the driver seat. Macros are simply more of that.
Implementing macros makes your brain melt. However, using them is relatively straightforward.
December 23, 2012 17:07
December 22, 2012
So it has come to this. Threads.pm is up and running, bringing the ever-wanted threaded execution to the most popular Perl 6 implementation.
You’re looking for TL;DR, aren’t you? Here’s what it’s capable of:
use Threads;
use Semaphore;
my @elements;
my $free-slots = Semaphore.new(value => 10);
my $full-slots = Semaphore.new(value => 0);
sub produce($id) {
my $i = 0;
loop {
$free-slots.wait;
@elements.push: $i;
$i++;
$full-slots.post;
}
}
sub consume($id) {
loop {
$full-slots.wait;
my $a = @elements.shift;
$free-slots.post;
}
}
for 1..5 -> $i {
async sub { produce($i) }
}
for 5..10 -> $i {
async sub { consume($i) }
}
Doesn’t look that awesome. I mean, it’s just a producer-consumer problem, what’s the big deal? Let me repeat:
OMG, RAKUDO HAS WORKING THREADS.
So, once we’re done celebrating and dancing macarena all around, there’ll always be someone to ask “hold on, there’s gotta be a caveat. Something surely is missing, explain yourself!”
I’ll be delighted to say “nope, everything’s there!”, but that’d make me a liar. Yeah, there are missing pieces. First, those aren’t really native threads – just green threads. Native OS threads are already implemented in Parrot VM, but NQP (the language that Rakudo is based on) still doesn’t support them, so before some volunteer comes along to fix them, you’ll still have to build parrot --without-threads (which means: use green threads, not OS threads) for Threads.pm to work. But fear not! The API is exactly the same, so once native threads are there, both Threads.pm and the code you write with it should work without any changes.
But green threads are fine too! Except for one minor detail: whenever any of them blocks on IO, the entire Parrot comes to a halt. The plan is for Parrot threads scheduler to handle it nicely, but it’s not there yet, so if you expected nice and easy async IO, sorry, but you’re stuck on MuEvent :)
Yep, we’re not really there yet. But I claim it’s closer than ever. We have working threads implementation. You can write code with that, and it’s not a PITA. Go for it! There’s a lot of room to improve it. I didn’t try really hard for Threads.pm to follow the concurrency synopsis (in my defense, I think niecza doesn’t follow it either :)), and I think that once we unleash a wolfpack of developers which can work towards something that we’ll all love to use.
December 22, 2012 10:36
TODO: Insert the description of changes. Also, mention that Rakudo 2012.12 (Warszawa) was released and that Rakudo isn’t Rakudo Star.
Incompatible changes
Rakudo Perl (in Rakudo 2012.12)
- When disambiguating hash/block, and
$_ variable is used, Rakudo assumes block. This could break your code if you were expecting a hash.
Rakudo Perl (post Rakudo 2012.12)
my $a; { $a; my $a } is an error.for runs in sink (void) or list context, just like map (actually, for does exactly what map does). If you were depending on it, you can use eager do before for to force eager execution.
New features
Rakudo Perl (in Rakudo 2012.12)
- Added
X::Comp::Group exception. It’s used when compilation can produce multiple issues.
- Added support for warnings. Code like
my ($a, $a); only causes warnings, not fatal errors.
- Added
X::Undeclared::Symbols exception for variables that weren’t declared, but used.
- You can use
.message on Exceptions.
- Fixed
.perl output for a Pair that has key that is a Pair.
X::Item is thrown for @a[NaN] or @a[Inf]. It isn’t incompatible change because it wasn’t working before anyway.
December 22, 2012 08:00
Guest post by Herbert Breunung (lichtkind).
Perl 5 brought regexes to mainstream programming and set a standard, one that is felt as relevant even in Redmond. Perl 6, of course, steps up the game by adding many new features to the regex camp, including easy-to-build grammars for your own complex parsers. But without getting too complex, you can get a lot of joy out of Perl 6′s rx (that’s how Perl 6 spells Perl 5′s qr operator, that enables you to save a Regex in a variable).
Because the Perl 6 regex syntax is less littered with exceptional cases, Larry Wall also likes to joke that he put the “regular” back into “regular expression”.
Some of the changes are:
- most special variables are gone,
- non-capturing groups and other grouping syntax is easier to type,
- no more single/multi line modes,
- x mode became default, making whitespace non-significant by default.
In summary, regexes are more regular than in Perl 5, confirming Larry’s joke. They try a bit harder to make your life easier when you need to match text. Under the hood, regexes have blossomed out into a complete sub-language within the bigger Perl 6 language. A language with its own parsing rules.
But don’t fret; not everything has changed. Some things remain the same:
/\d+/
This regex still matches one or more consecutive digits.
Similarly, if you want to capture the digits, you can do this, just like you’re used to:
/(\d+)/
You’ll find the matched digits in $0, not $1 as in Perl 5. All the special variables $0, $1, $2 are really syntactic sugar for indexing the match variable ($/[0], $/[1], $/[2]). Because indices start at 0, it makes sense for the first matched group to be $0. In Perl 5, $0 contains the name of the script or program, but this has been renamed into $*EXECUTABLE_NAME in Perl 6.
Should you be interested in getting all of the captured groups of a regex match, you can use @(), which is syntactic sugar for @($/).
The object in the $/ variable holds lots of useful information about the last match. For example, $/.from will give you the starting string position of the match.
But $0 will get us far enough for this post. We use it to extract individual features from a string.
Sometimes we want to extract a whole bunch of similar things at once. Then we can use the :g (or :global) modifier on the regex:
$_ = '1 23 456 78.9';
say .Str for m:g/(\d+)/; # 1 23 456 78 9
Note that the :g — as opposed to prior regex implementations — sits up front, right at the start of the regex. Not at the end. That way, when you read the regex from left to right, you will know from the start how the regex is doing its matching. No more end-heavy regex expressions.
Matching “all things that look like this” is so useful, that there’s even a dedicated method for that, .comb:
$str.comb(/\d+/);
If you’re familiar with .split, you can think of .comb as its cheerful cousin, matching all the things that .split discards.
Let’s tackle the matching of an IPv4 address. Coming from a Perl 5 angle, we expect to have to do something like this:
/(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/
This won’t do in Perl 6, though. First of all, the {} blocks are real blocks in a Perl 6 regex; they contain Perl 6 code. Second, because Perl 6 has lots of error handling to catch p5isms, like this, you’ll get an error saying “Unsupported use of {N,M} as general quantifier; in Perl 6 please use ** N..M (or ** N..*)”.
So let’s do that. To match between one and three digits in a Perl 6 regex, we should type:
/\d ** 1..3/
Note how the regex sublanguage re-uses parts from the main Perl 6 language. ** can be seen as a kind of exponentiation (if we squint), in that we’re taking \d “to the between-first-and-third power”. And the range notation 1..3 exists both outside and within regexes.
Using our new knowledge about the repetition quantifier, we end up with something like this:
/(\d**1..3) \. (\d**1..3) \. (\d**1..3) \. (\d**1..3)/
That’s still kinda clunky. We might end up wishing that we could use the repetition operator again, but those literal dots in between prevent us from doing that. If only we could specify repetition a given number of times and a divider.
In Perl 6 regexes, you can.
/ (\d ** 1..3) ** 4 % '.' /
The % operator here is a quantifier modifier, so it can only follow on a quantifier like * or + or **. The choice of % for this function is relatively new in Perl 6, and you may prefer to read it as “modulo”, just like in the main language. That is, “match four groups of digits, modulo literal dots in between”. Or you could think of the dots in between as the “remainder”, the separators that are left after you’ve parsed the actual elements.
Oh, and you might’ve noticed that \. changed to '.' on the way. We can use either; they mean exactly the same. In Perl 5, there isn’t a simple rule saying which symbols have a magic meaning and which ones simply signify themselves. In Perl 6, it’s easy: word characters (alphanumerics and the underscore) always signify themselves. Everything else has to be escaped or quoted to get its literal meaning.
Putting it all together, here’s how we would extract IPv4 addresses out of a string:
$_ = "Go 127.0.0.1, I said! He went to 173.194.32.32.";
say .Str for m:g/ (\d ** 1..3) ** 4 % '.' /;
# output: 127.0.0.1 173.194.32.32
Or, we could use .comb:
$_ = "Go 127.0.0.1, I said! He went to 173.194.32.32.";
my @ip4addrs = .comb(/ (\d ** 1..3) ** 4 % '.' /);
If we’re interested in individual integers, we can get those too:
$_ = "Go 127.0.0.1, I said! He went to 173.194.32.32.";
say .list>>.Str.perl for m:g/ (\d ** 1..3) ** 4 % '.' /;
# output: ("127", "0", "0", "1") ("173", "194", "32", "32")
If you want to know more, read the S05, or watch me battling with my slide deck and the English language in this presentation about regexes.
December 22, 2012 00:01
December 21, 2012
The Collatz sequence is one of those interesting “simple” math problems that I’ve run into a number of times. Most recently a blog post on programming it in Racket showed up on Hacker News. As happens so often, I instantly wanted to implement it in Perl 6.
sub collatz-sequence(Int $start) {
$start, { when * %% 2 { $_ / 2 }; when * !%% 2 { 3 * $_ + 1 }; } ... 1;
}
sub MAIN(Int $min, Int $max) {
say [max] ($min..$max).map({ +collatz-sequence($_) });
}
This is a very straightforward implementation of the Racket post’s max-cycle-length-range as a stand-alone p6 script. collatz-sequence generates the sequence using the p6 sequence operator. Start with the given number. If it is divisible by two, do so: when * %% 2 { $_ / 2 }. If it is not, multiply by three and add 1: when * !%% 2 { 3 * $_ + 1 }. Repeat this until the sequence reaches 1.
MAIN(Int $min, Int $max) sets up our main function to take two integers. Many times I don’t bother with argument types in p6, but this provides a nice feedback for users:
> perl6 collatz.pl blue red
Usage:
collatz.pl <min> <max>
The core of it just maps the numbers from $min to $max (inclusive) to the length of the sequence (+collatz-sequence) and then says the max of the resulting list ([max]).
Personally I’m a big fan of using the sequence operator for tasks like this; it directly represents the algorithm constructing the Collatz sequence in a simple and elegant fashion. On the other hand, you should be able to memoize the recursive version for a speed increase. Maybe that would give it an edge over the sequence operator version?
Well, I was wildly wrong about that.
sub collatz-length($start) {
given $start {
when 1 { 1 }
when * !%% 2 { 1 + collatz-length(3 * $_ + 1) }
when * %% 2 { 1 + collatz-length($_ / 2) }
}
}
sub MAIN($min, $max) {
say [max] ($min..$max).map({ collatz-length($_) });
}
This recursive version, which makes no attempt whatsoever to be efficient, is actually better than twice as fast as the sequence operator version. In retrospect, this makes perfect sense: I was worried about the recursive version making a function call for every iteration, but the sequence version has to make two, one to calculate the next iteration and the other to check and see if the ending condition has been reached.
Well, once I’d gotten this far, I thought I’d better do things correctly. I wrote two framing scripts, one for timing all the available scripts, the other for testing them to make sure they work!
my @numbers = 1..200, 10000..10200;
sub MAIN(Str $perl6, *@scripts) {
my %results;
for @scripts -> $script {
my $start = now;
qqx/$perl6 $script { @numbers }/;
my $end = now;
%results{$script} = $end - $start;
}
for %results.pairs.sort(*.value) -> (:key($script), :value($time)) {
say "$script: $time seconds";
}
}
This script takes as an argument a string that can be used to call a Perl 6 executable and a list of scripts to run. It runs the scripts using the specified executable, and times them using p6′s now function. It then sorts the results into order and prints them. (A similar script I won’t post here tests each of them to make sure they are returning correct results.)
In the new framework, the Collatz script has changed a bit. Instead of taking a min and a max value and finding the longest Collatz sequence generated by a number in that range, it takes a series of numbers and generates and reports the length of the sequence for each of them. Here’s the sequence operator script in its full new version:
sub collatz-length(Int $start) {
+($start, { when * %% 2 { $_ / 2 }; when * !%% 2 { 3 * $_ + 1 }; } ... 1);
}
sub MAIN(*@numbers) {
for @numbers -> $n {
say "$n: " ~ collatz-length($n.Int);
}
}
For the rest of the scripts I will skip the MAIN sub, which is exactly the same in each of them.
Framework established, I redid the recursive version starting from the new sequence operator code.
sub collatz-length(Int $n) {
given $n {
when 1 { 1 }
when * %% 2 { 1 + collatz-length($_ div 2) }
when * !%% 2 { 1 + collatz-length(3 * $_ + 1) }
}
}
The sharp-eyed will notice this version is different from the first recursive version above in two significant ways. This time I made the argument Int $n, which instantly turned up a bit of a bug in all implementations thus far: because I used $_ / 2, most of the numbers in the sequence were actually rationals, not integers! This shouldn’t change the results, but is probably less efficient than using Ints. Thus the second difference about, it now uses $_ div 2 to divide by 2. This version remains a great improvement over the sequence operator version, running in 4.7 seconds instead of 13.3. Changing when * !%% 2 to a simple default shaves another .3 seconds off the running time.
Once I started wondering how much time was getting eaten up by the when statements, rewriting that bit using the ternary operator was an obvious choice.
sub collatz-length(Int $start) {
+($start, { $_ %% 2 ?? $_ div 2 !! 3 * $_ + 1 } ... 1);
}
Timing results: Basic sequence 13.4 seconds. Sequence with div 11.5 seconds. Sequence with div and ternary 9.7 seconds.
That made me wonder what kind of performance I could get from a handcoded loop.
sub collatz-length(Int $n is copy) {
my $length = 1;
while $n != 1 {
$n = $n %% 2 ?? $n div 2 !! 3 * $n + 1;
$length++;
}
$length;
}
That’s by far the least elegant of these, I think, but it gets great performance: 3 seconds.
Switching back to the recursive approach, how about using the ternary operator there?
sub collatz-length(Int $n) {
return 1 if $n == 1;
1 + ($n %% 2 ?? collatz-length($n div 2) !! collatz-length(3 * $n + 1));
}
This one just edges out the handcoded loop, 2.9 seconds.
Can we do better than that? How about memoization? is cached is supposed to be part of Perl 6; neither implementation has it yet, but last year’s Advent calendar has a Rakudo implementation that still works. Using the last version changed to sub collatz-length(Int $n) is cached { works nicely, but takes 3.4 seconds to execute. Apparently the overhead of caching slows it down a bit. Interestingly, the non-ternary recursive version does speed up with is cached, from 4.4 seconds to 3.6 seconds.
Okay, instead of using a generic memoization, how about hand-coding one?
sub collatz-length(Int $n) {
return 1 if $n == 1;
state %lengths;
return %lengths{$n} if %lengths.exists($n);
%lengths{$n} = 1 + ($n %% 2 ?? collatz-length($n div 2) !! collatz-length(3 * $n + 1));
}
Bingo! 2.7 seconds.
I’m sure there are lots of other interesting approaches for solving this problem, and encourage people to send them in. In the meantime, here’s my summary of results so far:
The table was generated from timing-table-generator.pl.
December 21, 2012 12:25
December 20, 2012
Today, let’s talk about DSLs.
Post from the past: a motivating example
Two years ago I wrote a blog post about Nim, a game played with piles of stones. I just put in ASCII diagrams of the actual Nim stone piles, telling myself that if I had time, I would put in fancy SVG diagrams, generated with Perl 6.
Naturally, I didn’t have time. My self-imposed deadline ran out, and I published the post with simple ASCII diagrams.
But time is ever-regenerative, and there for people who want it. So, let’s generate some fancy SVG diagrams with Perl 6.
Have bit array, want SVG
What do we need, exactly? Well, a subroutine that takes an array of piles as input and generates an SVG file would be a really good start.
Let’s take the last “image” in the post as an example:
3 OO O
4 OOOO
5 OOOO O
For the moment, let’s ignore the numbers at the left margin; they’re just counting stones. We summarize the piles themselves as a kind of bitmap, which also forms the input to the function:
my @piles =
[0, 0, 0, 0, 0, 0, 0, 0, 1],
[1, 1, 1, 1, 0, 1, 1, 0, 1],
[1, 1, 1, 1, 0, 0, 0, 0, 1];
nim-svg(@piles);
At this point, we need only create the nim-svg function itself, and make it render SVG from this bitmap. Since I’ve long since tired of outputting SVG by hand, I use the SVG module, which comes bundled with Rakudo Star.
use SVG;
sub nim-svg(@piles) {
my $width = max map *.elems, @piles;
my $height = @piles.elems;
my @elements = gather for @piles.kv -> $row, @pile {
for @pile.kv -> $column, $is_filled {
if $is_filled {
take 'circle' => [
:cx($column + 0.5),
:cy($row + 0.5),
:r(0.4)
];
}
}
}
say SVG.serialize('svg' => [ :$width, :$height, @elements ]);
}
I think you can follow the logic in there. The subroutine simply iterates over the bitmap, turning 1s into circles with appropriate coordinates.
That’s it?
Well, this will indeed generate an SVG image for us, with the stones correctly placed. But let’s look again at the input that helped create this image:
[0, 0, 0, 0, 0, 0, 0, 0, 1],
[1, 1, 1, 1, 0, 1, 1, 0, 1],
[1, 1, 1, 1, 0, 0, 0, 0, 1];
Clearly, though we can discern the stones and gaps in there if we squint in a bit-aware programmer’s fashion, the input isn’t… visually attractive. (The zeroes even look like stones, even though they’re gaps!)
We can do better
Instead of using a bit array, let’s start from the desired SVG image and try to make the input look like that.
So, this is what I would prefer to write instead of a bitmask:
nim {
_ _ _ _ _ _ _ _ o;
o o o o _ o o _ o;
o o o o _ _ _ _ o;
}
That’s better. That looks more like my original ASCII diagram, while still being syntactic Perl 6 code.
Making a DSL
Wikipedia talks about a DSL as a language “dedicated to a particular problem domain”. Well, the above way of specifying the input would be a DSL dedicated to solving the draw-SVG-images-of-Nim-positions domain. (Admittedly a fairly narrow domain. But I’m mostly out to show the potential of DSLs in Perl 6, not to change the world with this particular DSL.)
Now that we have the desired end state, how do we connect the wires and make the above work? Clearly we need to declare three subroutines: nim, _, o. (Yes, you can name a subroutine _, no sweat.)
sub nim(&block) {
my @*piles;
my @*current-pile;
&block();
finish-last-pile();
nim-svg(@*piles);
}
sub _(@rest?) {
unless @rest {
finish-last-pile();
}
@*current-pile = 0, @rest;
return @*current-pile;
}
sub o(@rest?) {
unless @rest {
finish-last-pile();
}
@*current-pile = 1, @rest;
return @*current-pile;
}
Ok… explain, please?
A couple of things are going on here.
- The two variables
@*piles and @*current-pile are dynamic variables which means that they are visible not just in the current lexical scope, but also in all subroutines called before the current scope has finished. Notably, the two subroutines _ and o.
- The two subroutines
_ and o take an optional parameter. On each row, the rightmost _ or o acts as a silent “start of pile” marker, taking the time to do a bit of bookkeeping with the piles, storing away the last pile and starting on a new one.
- Each row in the DSL-y input basically forms a chain of subroutine calls. We take this into account by both incrementally building the
@*current-pile array at each step, all the while returning it as (possible) input for the next subroutine call in the chain.
And that’s it. Oh yeah, we need the bookkeeping routine finish-last-pile, too:
sub finish-last-pile() {
if @*current-pile {
push @*piles, [@*current-pile];
}
@*current-pile = ();
}
So, it works?
Now, the whole thing works. We can turn this DSL-y input:
nim {
_ _ _ _ _ _ _ _ o;
o o o o _ o o _ o;
o o o o _ _ _ _ o;
}
…into this SVG output:
<svg
xmlns="http://www.w3.org/2000/svg"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink"
width="9" height="3">
<circle cx="8.5" cy="0.5" r="0.4" />
<circle cx="0.5" cy="1.5" r="0.4" />
<circle cx="1.5" cy="1.5" r="0.4" />
<circle cx="2.5" cy="1.5" r="0.4" />
<circle cx="3.5" cy="1.5" r="0.4" />
<circle cx="5.5" cy="1.5" r="0.4" />
<circle cx="6.5" cy="1.5" r="0.4" />
<circle cx="8.5" cy="1.5" r="0.4" />
<circle cx="0.5" cy="2.5" r="0.4" />
<circle cx="1.5" cy="2.5" r="0.4" />
<circle cx="2.5" cy="2.5" r="0.4" />
<circle cx="3.5" cy="2.5" r="0.4" />
<circle cx="8.5" cy="2.5" r="0.4" />
</svg>
Yay!
Summary
The principles I used in this post are fairly easy to generalize. Start from your desired DSL, and create the subroutines to make it happen. Have dynamic variables handle the communication between separate subroutines.
DSLs are nice because they allow us to shape the code we’re writing around the problem we’re solving. Using relatively little “adapter code”, we’re left to focus on describing and solving problems in a natural way, making the programming language rise to our needs instead of lowering ourselves down to its needs.
December 20, 2012 00:01
December 19, 2012
On behalf of the Parrot team, I'm proud to announce Parrot 4.11.0, also known
as "All together - Happy Birthday Lovebird".
Parrot (http://parrot.org/) is a virtual machine aimed at running all
dynamic languages.
Parrot 4.11.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/4.11.0/),
or by following the download instructions at http://parrot.org/download.
For those who would like to develop on Parrot, or help develop Parrot
itself, we recommend using
Git to retrieve the source code to get the latest and best Parrot code.
Threads are considered experimental. There are a known GC bugs when
running large threads
on darwin and sometimes linux. There are no known problems when
running large single-threaded programs.
Reading from shared variables is transparent, writing must be done via
scheduled updater subs
on the owner thread.
You can use the API without using native OS threads by configuring
--without-threads.
Parrot 4.11.0 News:
- Core
+ Parrot now supports fast and lightweight OS threads, based
on Nat Tucks's
initial GSoC work on green threads and then Stefan Seifert's extension
to true parallel OS threads as hybrid threads.
See http://wknight8111.blogspot.co.at/2010/08/gsoc-threads-chandons-results.html
and http://niner.name/Hybrid_Threads_for_the_Parrot_VM.pdf
Summary:
http://perl6advent.wordpress.com/2012/12/11/day-11-parrot-threads/
+ New parrot cmdline option --numthreads
+ Export new dynpmc_class_<ClassName> types [GH #870].
+ Changed core_ops library format and ops2c to contain the
PBC_COMPAT only,
not the parrot version triple. Removed the duplicate
PARROT_DYNOP_CORE_INIT
cpp definition in include/parrot/oplib/core_ops.h, use only
PARROT_CORE_OPLIB_INIT from config.h
+ Store PBC_MAJOR and PBC_MINOR in the config hash, stored by
auto::revision
+ Added getprotobyname method to the Socket PMC
- Documentation
+ Marked unused API parameters
+ Install man pages for all binaries
- Languages
+ nqp-rx tests updated from perl6/nqp-rx.git, and enabled to
run on make test
- Platforms
+ More compilation fixes on Windows without working mt.exe or
ranlib=rem,
enable smoking without git binary.
- Tools
+ Re-enabled pbc_dump -h --header-only
+ Improved pmc2c, added more UNUSED(arg) macros [GH #836].
Parrot is now almost
-Wunused-parameters safe.
- Tests
+ failing Pod::Simple pod_todo.t tests are now skipped
+ added nqp-rx tests
+ removed duplicate ext/nqp-rx/t/p6regex tests
+ rewrote and enabled native_pbc tests [GH #394],
renamed native_pbc testfiles
The SHA256 message digests for the downloadable tarballs are:
46de72022dfe521fb8c5d95453e1d091d4d20d8f7ca4f2ed0cd1420b682f20dc
parrot-4.11.0.tar.gz
0632ad0ca9557df391088bddbbc2608a4503adb5679dcbe62c065a73d4ce49de
parrot-4.11.0.tar.bz2
Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next supported release is 15 January 2013.
Enjoy!
--
Reini Urban
http://cpanel.net/ http://www.perl-compiler.org/
December 19, 2012 01:49
Today I’ll write about coroutines, gather-take and why they are as much fun as one another. But since it’s all about manipulating control flow, I took the liberty to reorganize the control flow of this advent post, so coroutines will finally appear somewhere at the end of it. In the meantime I’ll introduce the backstory, the problems that coroutines solved and how it looks from the Perl 6 kitchen.
LWP::Simple is all fun and games, but sometimes you can’t afford to wait for the result to come. It would make sense to say “fetch me this webpage and drop me a note when you’re done with it”. That’s non trivial though; LWP::Simple is a black box, which we tell “get() this, get() that” and it gives us the result back. There is no possible way to intercept the internal data it sends there and around. Or is there?
If you look at Perl 5′s AnyEvent::HTTP, you’ll see that it reimplemented the entire HTTP client to have it non-blocking. Let’s see if we can do better than that.
First thing, where does LWP::Simple actually block? Behind our backs it uses the built-in IO::Socket::INET class. When it wants data from it, it calls .read() or .recv() and patiently waits until they’re done. If only we could somehow make it not rely on those two directly, hmm…
„I know!”, a gemstone-fascinated person would say, „We can monkey-patch IO::Socket::INET”. And then we have two problems. No, we’ll go the other way, and follow the glorious path of Dependency Injection.
That sounds a bit scary. I’ve heard about as many definitions of Dependency Injection as many people I know. The general idea is to not create objects inside other objects directly; it should be possible to supply them from the outside. I like to compare it to elimination of „magic constants”. No one likes those; if you think of classes as another kind of magic constants which may appear in somebody else’s code, this is pretty much what this is about. In our case it looks like this:
# LWP::Simple make_request
my IO::Socket::INET $sock .= new(:$host, :$port);
There we go. “IO::Socket::INET” is the magic constant here; if you want to use a different thing, you’re doomed. Let’s mangle it for a bit and allow the socket class to come from the outside.
We’ll add an attribute to LWP::Simple, let’s call it $!socketclass
has $.socketclass = IO::Socket::INET;
If we don’t supply any, it will just fallback to IO::Socket::INET, which is a sensible default. Then, instead of the previous .new() call, we do
my $sock = $!socketclass.new(:$host, :$port);
The actual patch (https://github.com/tadzik/perl6-lwp-simple/commit/93c182ac2) is a bit more complicated, as LWP::Simple supports calling get() not only on constructed objects but also on type objects, which have no attributes set, but we only care about the part shown above. We have an attribute $!socketclass, which defaults to IO::Socket::INET but we’re free to supply another class – dependency-inject it. Cool! So in the end it’ll look like this:
class Fakesocket is IO::Socket::INET {
method recv($) {
note 'We intercepted recv()';
callsame;
}
method read($) {
note 'We intercepted read()';
callsame;
}
}
# later
my $lwp = LWP::Simple.new(socketclass => Fakesocket);
And so our $lwp is a fine-crafted LWP::Simple which could, theorically, give the control flow back to us while it waits for read() and recv() to finish. So, how about we put theory into practice?
Here start the actual coroutines, sorry for being late :)
What do we really need in our modified recv() and read()? We need a way to say „yeah, if you could just stop executing and give time to someone else, that would be great.” Oh no, but we have no threads! Luckily, we don’t need any. Remember lazy lists?
my @a := gather { for 1..* -> $n { take $n } }
So on one hand we run an infinite for loop, and on the other we have a way to say „give back what you’ve come up with, I’ll catch up with you later”. That’s what take() does: it temporarily jumps out of the gather block, and is ready to get back to it whenever you want it. Do I hear the sound of puzzles clicking together? That’s exactly what we need! Jump out of the execution flow and wait until we’re asked to continue.
class Fakesocket is IO::Socket::INET {
method recv($) {
take 1;
callsame;
}
method read($) {
take 1;
callsame;
}
}
# later
my @a := gather {
$lwp.get("http://jigsaw.w3.org/HTTP/300/301.html");
take "done";
}
# give time to LWP::Simple, piece by piece
while ~@a.shift ne "done" {
say "The coroutine is still running"
}
say "Yay, done!";
There we go! We just turned LWP::Simple into a non-blocking beast, using almost no black magic at all! Ain’t that cool.
We now know enough to create some syntactic sugar around it all. Everyone likes sugar.
module Coroutines;
my @coroutines;
enum CoroStatus <still_going done>;
sub async(&coroutine) is export {
@coroutines.push($(gather {
&coroutine();
take CoroStatus::done;
}));
}
#= must be called from inside a coroutine
sub yield is export {
take CoroStatus::still_going;
}
#= should be called from mainline code
sub schedule is export {
return unless +@coroutines;
my $r = @coroutines.shift;
if $r.shift ~~ CoroStatus::still_going {
@coroutines.push($r);
}
}
We maintain a list of coroutines currently running. Our async() sub just puts a block of code in the execution queue. Then every call to yield() will make it jump back to the mainline code. schedule(), on the other hand, will pick the first available coroutine to be run and will give it some time to do whatever it wants.
Now, let us wait for the beginning of the post to catch up.
December 19, 2012 00:01
December 18, 2012
Today, Perl turns 25: happy birthday Perl! There’s too much to say about this language, its philosophy, its culture, … So here, I would just thank all people who make Perl a success, for such a long time.
Introduction
A formula is “an entity constructed using the symbols and formation rules of a given language“, according to Wikipedia as of this writing. These words sound really familiar for any Perl 6 users who have already played with grammars, however this is not the purpose of this article. Instead, the aim is to demonstrate how the Perl 6 language can be easily extended in order to use formulas literally in the code.
There are many domains, like Mathematics, Physics, finance, etc., that use their own specific languages. When writing programs for such a domain, it could be less error-prone and simpler to use its specific language instead of using a specific API. For example, someone who has knowledge in electronic may find the formula below:
4.7kΩ ± 5%
far more understandable than the following piece of code:
my $factory MeasureFactory.getSharedInstance();
my $resistance = $factory.createMeasure(value => 4700,
unit => Unit::ohm,
precision => 5);
The formula 4.7kΩ ± 5% will be used all along this article as an example.
Symbol k: return a modified value
Let’s start with the simplest symbol: k. Basically this is just a multiplier placed after a numeric value. To make the Perl 6 language support this new operator, there’s no need to know much about Perl 6 guts: operators are just funny looking sub-routines:
sub postfix:<k> ($a) is tighter(&infix:<*>) { $a * 1000 }
This just makes 4.7k return 4.7 * 1000, for example. To be a little bit picky, such kind of multiplier should not be used without a unit (ex. Ω) and not be coupled to another multiplier (ex. μ). This would have made this article a little bit more complex, so this is left as an exercise to the reader :) Regarding the tighter trait, it is already well explained in three other articles.
Symbols %: return a closure
The next symbol is %: it is commonly used to compute a ratio of something, that’s why 5% shouldn’t naively be transformed into 0.05. Instead, it creates a closure that computes the given percent of whatever you want:
sub postfix:<%> ($a) is tighter(&infix:<*>) { * * $a / 100 }
It’s now possible to write $f = 5%; $f(42) or 5%(42) directly, and this returns 2.1. It is worth saying this doesn’t conflict with the infix:<%> operator (modulo), that is, 5 % 42 still returns 5.
Symbol Ω: create a new Measure object
Let’s go on with the Ω symbol. One possibility is to tie the unit and the value in the same object, as in the Measure class defined below. The ACCEPTS method is explained later but the idea in this case is that two Measure objects with two different units can’t match together:
enum Unit <volt ampere ohm>;
class Measure {
has Unit $.unit;
has $.value;
method ACCEPTS (Measure:D $a) {
$!unit == $a.unit && $!value.ACCEPTS($a.value);
}
}
Then, one operator per unit can be defined in order to hide the underlying API, that is, to allow 4.7kΩ as an equivalent of Measure.new(value => 4.7k, unit => ohm):
sub postfix:<V> (Real:D $a) is looser(&postfix:<k>) {
Measure.new(value => $a, unit => volt)
}
sub postfix:<A> (Real:D $a) is looser(&postfix:<k>) {
Measure.new(value => $a, unit => ampere)
}
sub postfix:<Ω> (Real:D $a) is looser(&postfix:<k>) {
Measure.new(value => $a, unit => ohm)
}
Regarding the ACCEPTS method, it is used by ~~, the smartmatch operator, to check if the left operand can match the right operand, the one with the ACCEPTS method. In other terms, $a ~~ $b is equivalent to $b.ACCEPTS($a). Typically, this allows the intuitive comparison between two different types, like scalars and containers for example.
In this example, this method is overloaded to ensure two Measure objects can match only if they have the same unit and if their values match. That means 4kΩ ~~ 4.0kΩ is True whereas 4kΩ ~~ 4kV is False. Actually, there are many units that can mix altogether, typically currencies (¥€$) and the ones derived from the International System of Unit. But as usual, when something is a little bit more complex, it is left as an exercise to the reader ;)
Symbol ±: create a Range object
There’s only one symbol left so far: ±. In the example, it is used to indicate the tolerance of the resistance. This tolerance could be either absolute (expressed in Ω) or relative (expressed in %), thus the new infix:<±> operator has several signatures and have to be declared with a multi keyword. In both cases, the value is a new Range objects with the right bounds:
multi sub infix:<±> (Measure:D $a, Measure:D $b) is looser(&postfix:<Ω>) {
die if $a.unit != $b.unit;
Measure.new(value => Range.new($a.value - $b.value,
$a.value + $b.value),
unit => $a.unit);
}
multi sub infix:<±> (Measure:D $a, Callable:D $b) is looser(&postfix:<Ω>) {
Measure.new(value => Range.new($a.value - $b($a.value),
$a.value + $b($a.value)),
unit => $a.unit);
}
Actually, any Callable object could be used in the second variant, not only the closures created by the % operators.
So far, so good! It’s time to check in the Perl6 REPL interface if everything works fine:
> 4.7kΩ ± 1kΩ
Measure.new(unit => Unit::ohm, value => 3700/1..5700/1)
> 4.7kΩ ± 5%
Measure.new(unit => Unit::ohm, value => 4465/1..4935/1)
It looks good, so all the code above ought to be moved into a dedicated module in order to be re-used at will. Then, a customer could load it and write literally:
my $resistance = 4321Ω;
die "resistance is futile" if !($resistance ~~ 4.7kΩ ± 5%);
As of this writing, this works both in Niecza and Rakudo, the two most advanced implementations of Perl 6.
Symbols that aren’t operators
Symbols in a formula are not always operators, they can be symbolic constants too, like π. In many languages, constants are just read-only variables, which sounds definitely weird: a variable isn’t supposed to be … variable? In Perl 6, a constant can be a read-only variable too (hmm) or a read-only term (this sounds better). For example, to define the constant term φ:
constant φ = (1 + sqrt(5)) / 2;
Conclusion
In this article the Perl 6 language was slightly extended with several new symbols in order to embed simple formulas. Although it is possible to go further by changing the Perl 6 grammar in order to embed more specific languages, that is, languages that don’t have the same grammar rules. Indeed, there are already two such languages supported by Perl 6: regexp and quotes. The same way, Niecza use a custom language to connect its portable parts to the unportable.
Bonus: How to type these exotic symbols?
Most of the Unicode symbols can be type in Xorg — the most used interface system on Linux — thanks to the Compose key, also named Multi key. When this special key is pressed, all the following key-strokes are somewhat merged in order to compose a symbol.
There’s plenty of documentation about this support elsewhere on Internet, so only the minimal information is provided here. First, to map the Compose key to the Caps Lock key, write in a X terminal:
sh> setxkbmap -option compose:caps
Some compositions are likely already defined, for instance <caps> followed by + then - should now produce ±, but both Ω and φ are likely not defined. One solution is to write a
~/.XCompose file with the following content:
include "%L" # Don't discard the current locale setting.
<Multi_key> <o> <h> <m> : "Ω" U03A9
<Multi_key> <O> <underscore> : "Ω" U03A9
<Multi_key> <underscore> <O> : "Ω" U03A9
<Multi_key> <p> <h> <y> : "φ" U03C6
<Multi_key> <o> <bar> : "φ" U03C6
<Multi_key> <bar> <o> : "φ" U03C6
This takes effect for each newly started applications. Feel free to leave a comment if you know how to add such a support on other
systems.
December 18, 2012 00:01
December 17, 2012
Many people have heard of Perl 6, especially in the greater Perl community. However, Perl 6 has a complicated ecosystem which can be a littled daunting, so as a newcomer to the Perl 6 community myself, I thought I would share what I’ve learned.
How do I install Perl 6?
It’s simple; you can just download one of the existing implementations of the language (as Perl 6 is a specification), build it, and install it! There are several implementations out there right now, in various states of completion. Rakudo is an implementation that targets Parrot, and is the implementation that I will discuss most in this post. Niecza is another implementation that targets the CLR (the .NET runtime). For more information on these implementations and on other implementations, please see Perl 6 Compilers. Perl 6 is an ever-evolving language, and any compiler that passes the official test suite can be considered a Perl 6 implementation.
You mentioned “Parrot”; what’s that?
Parrot is a virtual machine that is designed to run dynamically typed languages. Along with the virtual machine, it includes tools for generating virtual machine code from intermediate languages (named PIR and PASM), as well as a suite of tools to make writing compilers easier.
What is Rakudo written in?
Rakudo itself is written primarly in Perl 6, with some bits of C for some of the lower-level operations, like binding method arguments and adding additional opcodes to the Parrot VM. It may seem strange to implement a Perl 6 compiler in Perl 6 itself; Rakudo uses NQP for building itself.
What’s NQP?
NQP (or Not Quite Perl 6) is an implementation of Perl 6 that is focused on creating compilers for the Parrot Compiler Toolkit. It is currently focused on targetting Parrot, but in the future, it may support various compilation targets, so you will be able to use Rakudo to compile your Perl 6 programs to Parrot opcodes, a JVM class file, or perhaps Javascript so you can run it in the browser. NQP is written in NQP, and uses a pre-compiled version of NQP to compile itself.
I hope that this information was useful to you, dear reader, and that it helps to clarify the different pieces of the Perl 6 ecosystem. As I learn more about each piece, I intend to write blog posts that will hopefully help others to get started contributing to Perl 6!
-Rob
December 17, 2012 00:00
December 16, 2012
All the precedence men
As I was taking a walk today, I realized one of the reasons why I like Perl. Five as well as six. I often hear praise such as “Perl fits the way I think”. And I have that feeling too sometimes.
If I were the president (or prime minister, as I’m Swedish), and had a bunch of advisers, maybe some of them would be yes-men, trying to give me advice that they think I will want to hear, instead of advice that would be useful to me. Some languages are like that, presenting us with an incomplete subset of the necessary tools. The Perl languages, if they were advisers, wouldn’t be yes-men. They’d give me an accurate view of the world, even if that view would be a bit messy and hairy sometimes.
Which, I guess, is why Perl five and six are so often used in handling messy data and turning it into something useful.
To give a few specific examples:
- Perl 5 takes quotes and quoting very seriously. Not just strings but lists of strings, too. (See the
qw keyword.) Perl 6 does the same, but takes quoting further. See see the recent post on quoting.
- jnthn shows in yesterday’s advent post that Perl 6 takes compiler phases seriously, and allows us to bundle together code that belongs together conceptually but not temporally. We need to do this because the world is gnarly and running a program happens in phases.
- Grammars in Perl 6 are not just powerful, but in some sense honest, too. They don’t oversimplify the task for the programmer, because then they would also limit the expressibility. Even though grammars are complicated and intricate, they should be, because they describe a process (parsing) that is complicated and intricate.
Operators
Perl is known for its many operators. Some would describe it as an “operator-oriented” language. Where many other language will try to guess how you want your operators to behave on your values, or perhaps demand that you pre-declare all your types so that there’ll be no doubt, Perl 6 carries much of the typing information in its operators:
my $a = 5;
my $b = 6;
say $a + $b; # 11 (numeric addition)
say $a * $b; # 30 (numeric multiplication)
say $a ~ $b; # "56" (string concatenation)
say $a x $b; # "555555" (string repetition)
say $a || $b; # 5 (boolean disjunction)
say $a && $b; # 6 (boolean conjunction)
Other languages will want to bunch together some of these for us, using the + operator for both numeric addition and string concatenation, for example. Not so Perl. You’re meant to choose yourself, because the choice matters. In return, Perl will care a little less about the types of the operands, and just deliver the appropriate result for you.
“The appropriate result” is most often a number if you used a numeric operator, and a string if you used a string operator. But sometimes it’s more subtle than that. Note that the boolean operators above actually preserved the numbers 5 and 6 for us, even though internally it treated them both as true values. In C, if we do the same, C will unhelpfully “flatten” these results down to the value 1, its spelling of the value true. Perl knows that truthiness comes in many flavors, and retains the particular flavor for you.
Operator precedence
“All operators are equal, but some operators are more equal than others.” It is when we combine operators that we realize that the operators have different “tightness”.
say 2 * 3 + 1; # 7, because (2 * 3) + 1
say 1 + 2 * 3; # 7, because 1 + (2 * 3), not 9
We can always be 100% explicit and surround enough of our operations with parentheses… but when we don’t, the operators seem to order themselves in some order, which is not just simple left-to-right evaluation. This ordering between operators is what we refer to as “precedence”.
No doubt you were taught in math class in school that multiplications should be evaluated before additions in the way we see above. It’s as if factors group together closer than terms do. The fact that this difference in precedence is useful is backed up by centuries of algebra notation. Most programming languages, Perl 6 included, incorporates this into the language.
By the way, this difference in precedence is found between other pairs of operators, even outside the realm of mathematics:
Additive (loose) Multiplicative (tight)
================ ======================
number + *
string ~ x
bool || &&
It turns out that they make as much sense for other types as they do for numbers. And group theory bears this out: these other operators can be seen as a kind of addition and multiplication, if we squint.
Operator precedence parser
Deep in the bowels of the Perl 6 parser sits a smaller parser which is very good at parsing expressions. The bigger parser which parses your Perl 6 program is a really good recursive-descent parser. It works great for creating syntax trees out of the larger program structure. It works less well on the level of expressions. Essentially, what trips up a recursive-descent parser is that it always has to create AST nodes for all the possible precedence levels, whether they’re present or not.
So this smaller parser is an operator-table parser. It knows what to do with each type of operator (prefix, infix, postfix…), and kind of weaves all the terms and operators into a syntax tree. Only the precedence levels actually used show up in the tree.
The optable parser works by comparing each new operator to the top operator on a stack of operators. So when it sees an expression like this:
$x ** 2 + 3 * $x - 5
it will first compare ** against + and decide that the former is tighter, and thus $x ** 2 should be put together into a small tree. Later, it compares + against *, and decides to turn 3 * $x into a small tree. It goes on like this, eventually ending up with this tree structure:
infix:<->
+-- infix:<+>
+-- infix:<**>
| +-- term:<$x>
| +-- term:<2>
+-- infix:<*>
+-- term:<3>
+-- term:<$x>
Because leaf nodes are evaluated first and the root node last, this tree structure determines the order of evaluation for the expression. The order ends up being the same as if the expression had these parentheses:
(($x ** 2) + (3 * $x)) - 5
Which, again, is what we’ve learned to expect.
Associativity
Another factor also governs how these invisible parentheses are to be distributed: operator associativity. It’s the concern of how the operator combines with multiple copies of itself, or other sufficiently similar operators on the same precedence level.
Some examples serve to explain the difference:
$x = $y = $z; # becomes $x = ($y = $z)
$x / $y / $z; # becomes ($x / $y) / $z
In both of these cases, we may look at the way the parentheses are doled out, and say “well, of course”. Of course we must first assign to $y and only then to $x. And of course we first divide by $y and only then by $z. So operators naturally have different associativity.
The optable parser compares not just the precedence of two operators but also, when needed, their associativity. And it puts the parentheses in the right place, just as above.
User-defined operators
Now we come back to Perl not being a yes-man, and working hard to give you the appropriate tools for the job.
Perl 6 allows you to define operators. See my post from last year on the details of how. But it also allows you to specify precedence and associativity of each new operator.
As you specify a new operator, a new Perl 6 parser is automatically constructed for you behind the scenes, which contains your new operator. In this sense, the optable parser is open and extensible. And Perl 6 gives you exactly the same tools for talking about precedence and associativity as the compiler itself uses internally.
Perl treats you like a grown-up, and expects you to make good decisions based on a thorough understanding of the problem space. I like that.
December 16, 2012 00:01
December 15, 2012
When writing programs, it’s important not only to separate the concerns that need separating, but also to try and keep related things close to each other. This gives the program a sense of cohesion, and helps to avoid the inevitable problems that arise when updating one part of a program demands an update in another far-away part. One especially tricky problem can be when the things we want to do are distributed over time. This can cause us to move related things apart in order to get them to happen at the times we want.
Phasers in Perl 6 help you keep related concepts together in your code, while also indicating that certain aspects of them should happen at different points during the lifetime of the current program, invocation or loop construct. Let’s take a look at some of them.
ENTER and LEAVE
One of the things I had most fun writing in Perl 6 recently was the debugger. There are various things that need a little care. For example, the debugger needs to look out for exceptions and, when they are thrown, give the user a prompt to let them debug why the exception was thrown. However, there is also a feature where, at the prompt, you can evaluate an expression. The debugger shouldn’t re-enter itself if this expression throws, so we need to keep track of if we’re already showing the prompt. This meant setting and clearing a flag. Thing is, the prompt method is relatively lengthy; it has a given/when to identify the various different commands. I could, of course, have set the prompt flag at the start and cleared it at the end. But that would have spread out the concern of maintaining the flag. Here’s what I did instead:
method issue_prompt($ctx, $cur_file) {
ENTER $in_prompt = True;
LEAVE $in_prompt = False;
# Lots of stuff here
}
This ensures the flag is set when we enter the method, cleared when we leave the method – and lets me keep the two together.
INIT and END
We’re writing a small utility and want to log what happens as we run it. Time wise, we want to:
- Open the log file at the start of the program, creating it if needed and overwriting an existing one otherwise
- Write log entries at various points during the program’s execution
- Close the log file at the end
Those three actions are fairly spread out in time, but we’d like to collect them together. This time, the INIT and END phasers come to the rescue.
sub log($msg) {
my $fh = INIT open("logfile", :w);
$fh.say($msg);
END $fh.close;
}
Here, we use INIT to perform an action at program start time. It turns out that INIT also keeps around the value produced by the expression following it, meaning it can be used as an r-value. This means we have the file handle available to us, and can write to it during the program. Then, at the END of the program, we close the file handle. All of these have block forms, should you wish to do something more involved:
sub log($msg) {
my $fh = INIT open("logfile", :w);
$fh.say($msg);
END {
$fh.say("Ran in {now - INIT now} seconds");
$fh.close;
}
}
Note the second use of INIT in this example, to compute and remember the program start time so we can use it in the subtraction later on.
FIRST, NEXT and LAST
These phasers work with loops. They fire the first time the loop body executes, at the end of every loop body execution, and after the last loop body execution. FIRST and LAST are especially powerful in so far as they let us move code that wants to special-case the first and last time the loop body runs inside of the loop construct itself. This makes the relationship between these bits of code and the loop especially clear, and lessens the chance somebody moves or copies the loop and forgets the related bits it has.
As an example, let’s imagine we are rendering a table of scores from a game. We want to write a header row, and also do a little ASCII art to denote the start and end of the table. Furthermore, we’d like to keep track of the best score each time around the loop, and then at the end print out the best score. Here’s how we could write it.
for %scores.kv -> $player, $score {
FIRST say "Score\tPlayer";
FIRST say "-----\t------";
LAST say "-----\t------";
NEXT (state $best_score) max= $score;
LAST say "BEST SCORE: $best_score";
say "$score\t$player";
}
Notice how we keep the header/footer code together, as well as being able to keep the best score tracking code together. It’s also all inside the loop, making its relationship to the loop clear. Note how the state variable also comes in useful here. It too is a construct that lets us keep a variable scoped inside a block even if its usage spans multiple invocations of the block.
KEEP and UNDO
These are variants of LEAVE that trigger conditional on the block being successful (KEEP) or not (UNDO). A successful block completes without unhandled exceptions and returns a defined value. An unsuccessful block exits due to an exception or because it returns an undefined value. Say we were processing a bunch of files and want to build up arrays of successful files and failed files. We could write something like:
sub process($file) {
KEEP push @success, $file;
UNDO push @failure, $file;
my $fh = open($file);
# ...
}
There are probably a bunch of transaction-like constructs that can also be very neatly implemented with these two.
And there’s more!
While I’ve covered a bunch of the phasers here, there are some others. For example, there’s also BEGIN, which lets you do some computation at compile time. Hopefully, though, this set of examples gives you some inspiration in how phasers can be used effectively, as well as a better grasp of the motivation for them. Bringing related things together and setting unrelated things apart is something we need to think carefully about every day as developers, and phasers help us keep related concerns together, even if they should take place at different phases of our program’s execution.
December 15, 2012 13:28
So, this is yet another article in Perl 6 Changes series. (I know, those introductions are really boring, but would you like to see Lorem Ipsum Dolor instead)? As in previous week, Rakudo mainly improves Perl 6 parsing.
By the way, if you don’t know about Perl 6 Coding Contest, check it out. The challenges are rather easy if you had any programming language experience (if you don’t know Perl 6, check out Using Perl 6 book). If you still have questions, people in #perl6 can help you.
Incompatible changes
Rakudo Perl
uniq has === semantics. It could break your programs if you for example were using uniq to remove identical arrays (previously they were converted to strings which did what you wanted, as long the strings in array didn’t contained space or you didn’t have array in array.
New features
Rakudo Perl
-
“Two terms in a row” error message.
-
Information about what was expected, if possible.
$ perl6 -e '2 2'
===SORRY!===
Two terms in a row
at -e:1
------> 2 ⏏2
expecting any of:
postfix
infix or meta-infix
infix stopper
statement end
statement modifier
statement modifier loop
-
for 1..10 { say so 9999 == any(1..2000) } is 30 times faster (huge junctions are faster).
-
Added Texas versions of the Set and Bag operators, used in Bags and Sets article in Perl 6 Advent Calendar.
-
Rakudo Perl now informs about obsolete use of . to concatenate strings.
-
Implemented sub term:<foo>. Now 2 + 2 == 5 can be true for sufficiently huge values of 2 (not like it should be).
-
qww now works.
Niecza Perl
slurp now works without arguments.
December 15, 2012 08:00
December 14, 2012
For the third year in a row, let's write some Perl 6 contest code!
The 2012 Perl 6 Coding Contest
Perl 6 has had a great year, with its two major implementations growing increasingly mature. Sometimes it's easy to forget all the progress that happens each month. Features and performance have improved all through the year. I think that will show in this year's entries.
So here's what's happening:
do five tasks, and you might win Amazon books worth €100!
Addendum: Thanks to an anonymous donor, we're proud to announce a second prize as well: Amazon books worth 100 USD! (Redeemable within a year of winning.)
The contest starts now, today on 2012-12-14. It ends five weeks later, on 2013-01-18. Registration is open for two weeks, starting now. Just send an email to cmasak@gmail.com — saying "sign me up!" or even sending your Amazon wishlist so I'll know which books to buy you if you win.
You might be curious about what you need to do to win. Here's an extract from the file rules.md:
Since "code quality" is a slightly subjective measure, let us provide a few
hints of what we'll be looking for:
* Correctness.
* Readability.
* Consistency.
* Clarity of intent.
* Algorithmic efficiency.
* Idiomatic use of Perl 6.
* Brevity.
In short, what we're looking for is top-quality code. That's how you win.
Here are the five tasks. Write Perl 6 programs to...
- Tell knights from knaves based on what they say. (more)
- Generate rectangle haikus. (more)
- Arrange wire crossings to rearrange wires. (more)
- Simulate rain in a world of cubes. (more)
- Distribute weights in bags evenly. (more)
I've chosen the problems so that they're easy to explain, but allow contestants
quite a bit of freedom to play around with various solutions.
This time, the whole contest is running as a GitHub repository. Easier than hosting zip files.
Now sign up — you've nothing to lose, and books to win! Also, this is your chance to finally take a look at Perl 6. 哈哈
(Looking for previous years' tasks and solutions? Here they are: 2010 tasks and 2010 solutions. 2011 tasks and 2011 solutions.)
December 14, 2012 20:21
Our brains are hard-wired to look for patterns, even where none exist. So, it’s no surprise that as soon as mankind started counting things, he would look for patterns in numbers. One group of numbers that have resisted the pattern matching capabilities of the human brain are the so-called “prime numbers”. These are numbers that can only be evenly divided by 1 or themselves–they have no other factors.
But you knew that already, so why am I talking about prime numbers instead of Perl 6? Because, just like our ancestors, the people that created Perl 6 and continue to shape it to be around for the next 100 years or more find prime numbers interesting. So interesting, in fact, that the language specification was modified to include a routine for determining whether or not a number is prime.
Alpha
At first, implmementations of this prime-number-finder were pure Perl 6 and took advantage of other features of the language such as ranges and junctions. An example implementation is shown below:
sub is-prime($n) { $n %% none 2..sqrt $n }
This implementation checks to see that none of numbers from 2 to the square root of $n will evenly divide $n. If this is the case, then the number is prime.
While the above implementation works fine, it is a little slow and it does suffer a little redundancy in the numbers it checks. For instance, if you know a number isn’t evenly divisible by 2, there’s no need to check if it’s evenly divisible by 4, yet the above algorithm does so anyway.
Beta
An improvement on the algorithm is to only check if the I between 2 and the square root of the number evenly divide the number. But … but … that’s like like defining a word in terms of itself. Thanks to ubiquitous lazy evaluation in Perl 6, that’s entirely possible. Here’s an implementation:
my @primes := 2, 3, 5, -> $p { ($p+2, $p+4 ... &is-prime)[*-1] } ... *;
sub is-prime($n) { $n %% none @primes ...^ * > sqrt $n }
The array @primes is an infinite, lazily evaluated sequence of numbers starting with 2, 3, and 5. The next number in the sequence is generated by creating a new sequence of odd numbers that start from the last odd number and continue until we reach a prime. That prime is the next number in the sequence. But how do we know if it’s a prime? We check with our handy C subroutine that actually uses the lazy list of primes up to the square root of the number we’re testing to see if any of them are factors.
There’s a kind of mutual recursion going on here where the @primes array effectively memoizes the primes we’ve seen so far. But … then there’s the problem that @primes will continue to grow as you check bigger and bigger numbers for prime-ness. Can we do better?
Indeed we can.
Gamma: Rabin-Miller test
Well … maybe we can. It depends on your idea of “better”. The Rabin-Miller primality test is probabalistic in nature. It doesn’t require storing an ever increasing cache of prime numbers to test if they are factors of the potential prime, but there is a chance that it will tell you that a number is prime when it actually isn’t. The good news is that we can adjust the odds so that we are reasonably confident that the number is prime. Here’s an implementation (taken from http://rosettacode.org/wiki/Miller-Rabin_primality_test#Perl_6):
sub expmod(Int $a is copy, Int $b is copy, $n) {
my $c = 1;
repeat while $b div= 2 {
($c *= $a) %= $n if $b % 2;
($a *= $a) %= $n;
}
$c;
}
subset PrimeCandidate of Int where { $_ > 2 and $_ % 2 };
my Bool multi sub is-prime(Int $n, Int $k) { return False; }
my Bool multi sub is-prime(2, Int $k) { return True; }
my Bool multi sub is-prime(PrimeCandidate $n, Int $k) {
my Int $d = $n - 1;
my Int $s = 0;
while $d %% 2 {(PrimeCandidate $n, Int $k)
$d div= 2;
$s++;
}
for (2 ..^ $n).pick($k) -> $a {
my $x = expmod($a, $d, $n);
next if $x == 1 or $x == $n - 1;
for 1 ..^ $s {
$x = $x ** 2 mod $n;
return False if $x == 1;
last if $x == $n - 1;
}
return False if $x !== $n - 1;
}
return True;
}
The third multi variant of is-prime with the signature (PrimeCandidate $n, Int $k) is where all of the magic happens. This multi is only triggered when the prime candidate ($n) is an odd number because of the definition of the PrimeCandidate type.
First, we factor out the powers of 2 from $n - 1. Since $n is an odd number, $n - 1 is even and so has at least one factor of 2. What we end up with is an odd number and some power-of-2 factors of $n - 1. We then use those factors to see if a random sample of $k numbers less than $n are congruent to the square roots of unity modulo $n (expmod handles the modular exponentiation). We repeat this for all of the powers of 2 we factored out of the original number. Fermat’s little theorem says that if we find any number where the congruence does not hold, then the number can not be prime.
The probability that this method will select a composite number as prime is based on how many numbers less than $n we choose to sample. If we select $k numbers to try, the probability is 4 ** -$k. By choosing to sample more numbers, we can quickly decrease the odds of a false positive to a negligible amount.
Wrap up
But … most people don’t really have to worry about the implementation details of is-prime. Not only have is-prime and expmod been added to the Perl 6 specification, but actual implementations (ala Rabin-Miller) have been added to the Rakudo and Niecza Perl 6 compilers. So, if you want to test your new cryptographic algorithm and need some large prime numbers, or if you’re developing a new random number generator and need some candidates for the modulus, or maybe you’re developing a new hashing algorithm, Perl 6 has a built-in is-prime that can help.
December 14, 2012 20:00
December 13, 2012
Over the years, I’ve written many variations on this code:
my %words;
for slurp.comb(/\w+/).map(*.lc) -> $word {
%words{$word}++;
}
(Aside: slurp.comb(/\w+/).map(*.lc) does the standard Perl trick of reading files specified on the command line or standard in, goes through the data for words, and makes them lowercase.)
Perl 6 introduces two new Associative types for dealing with this sort of functionality. KeyBag is drop-in replacement for Hash in this sort of case:
my %words := KeyBag.new;
for slurp.comb(/\w+/).map(*.lc) -> $word {
%words{$word}++;
}
Why would you prefer KeyBag over Hash in this case, considering that it’s a bit more code? Well, it does a better job of saying what you mean, if what you want is a positive Int-valued Hash. It actually enforces this as well:
> %words{"the"} = "green";
Unhandled exception: Cannot parse number: green
That’s Niecza’s error; Rakudo’s is less clear, but the important point is you get an error; Perl 6 detects that you’ve violated your contract and complains.
And KeyBag has a couple more tricks up its sleeve. First, four lines to initialize your KeyBag isn’t terribly verbose, but Perl 6 has no trouble getting it down to one line:
my %words := KeyBag.new(slurp.comb(/\w+/).map(*.lc));
KeyBag.new does its best to turn whatever it is given into the contents of a KeyBag. Given a List, each of the elements is added to the KeyBag, with the exact same result of our earlier block of code.
If you don’t need to modify the bag after its creation, then you can use Bag instead of KeyBag. The difference is Bag is immutable; if %words is a Bag, then %words{$word}++ is illegal. If immutability is okay for your application, then you can make the code even more compact:
my %words := bag slurp.comb(/\w+/).map(*.lc);
bag is a helper sub that just calls Bag.new on whatever you give it. (I’m not sure why there is no equivalent keybag sub.)
Bag and KeyBag have a couple more tricks up their sleeve. They have their own versions of .roll and .pick which weigh their results according to the given values:
> my $bag = bag "red" => 2, "blue" => 10;
> say $bag.roll(10);
> say $bag.pick(*).join(" ");
blue blue blue blue blue blue red blue red blue
blue red blue blue red blue blue blue blue blue blue blue
This wouldn’t be too hard to emulate using a normal Array, but this version would be:
> $bag = bag "red" => 20000000000000000001, "blue" => 100000000000000000000;
> say $bag.roll(10);
> say $bag.pick(10).join(" ");
blue blue blue blue red blue red blue blue blue
blue blue blue red blue blue blue red blue blue
They also work with all the standard Set operators, and have a few of their own as well. Here’s a simple demonstration:
sub MAIN($file1, $file2) {
my $words1 = bag slurp($file1).comb(/\w+/).map(*.lc);
my $words2 = set slurp($file2).comb(/\w+/).map(*.lc);
my $unique = ($words1 (-) $words2);
for $unique.list.sort({ -$words1{$_} })[^10] -> $word {
say "$word: { $words1{$word} }";
}
}
Passed two filenames, this makes a Bag from the words in the first file, a Set from the words in the second file, uses the set difference operator (-) to compute the set of words which are only in the first file, sorts those words by their frequency of appearance, and then prints out the top ten.
This is the perfect point to introduce Set. As you might guess from the above, it works much like Bag. Where Bag is a Hash from Any to positive Int, Set is a Hash from Any to Bool::True. Set is immutable, and there is also a mutable KeySet.
Between Set and Bag we have a very rich collection of operators:
| Operation |
Unicode |
“Texas” |
Result Type |
| is an element of |
∈ |
(elem) |
Bool |
| is not an element of |
∉ |
!(elem) |
Bool |
| contains |
∋ |
(cont) |
Bool |
| does not contain |
∌ |
!(cont) |
Bool |
| union |
∪ |
(|) |
Set or Bag |
| intersection |
∩ |
(&) |
Set or Bag |
| set difference |
|
(-) |
Set |
| set symmetric difference |
|
(^) |
Set |
| subset |
⊆ |
(<=) |
Bool |
| not a subset |
⊈ |
!(<=) |
Bool |
| proper subset |
⊂ |
(<) |
Bool |
| not a proper subset |
⊄ |
!(<) |
Bool |
| superset |
⊇ |
(>=) |
Bool |
| not a superset |
⊉ |
!(>=) |
Bool |
| proper superset |
⊃ |
(>) |
Bool |
| not a proper superset |
⊅ |
!(>) |
Bool |
| bag multiplication |
⊍ |
(.) |
Bag |
| bag addition |
⊎ |
(+) |
Bag |
Most of these are self-explanatory. Operators that return Set promote their arguments to Set before doing the operation. Operators that return Bag promote their arguments to Bag before doing the operation. Operators that return Set or Bag promote their arguments to Bag if at least one of them is a Bag or KeyBag, and to Set otherwise; in either case they return the type promoted to.
Please note that while the set operators have been in Niecza for some time, they were only added to Rakudo yesterday, and only in the Texas variations.
A bit of a word may be needed for the different varieties of unions and intersections of Bag. The normal union operator takes the max of the quantities in either bag. The intersection operator takes the min of the quantities in either bag. Bag addition adds the quantities from either bag. Bag multiplication multiplies the quantities from either bag. (There is some question if the last operation is actually useful for anything — if you know of a use for it, please let us know!)
> my $a = bag <a a a b b c>;
> my $b = bag <a b b b>;
> $a (|) $b;
bag("a" => 3, "b" => 3, "c" => 1)
> $a (&) $b;
bag("a" => 1, "b" => 2)
> $a (+) $b;
bag("a" => 4, "b" => 5, "c" => 1)
> $a (.) $b;
bag("a" => 3, "b" => 6)
I’ve placed my full set of examples for this article and several data files to play with on Github. All the sample files should work on the latest very latest Rakudo from Github; I think all but most-common-unique.pl and bag-union-demo.pl should work with the latest proper Rakudo releases. Meanwhile those two scripts will work on Niecza, and with any luck I’ll have the bug stopping the rest of the scripts from working there fixed in the next few hours.
A quick example of getting the 10 most common words in Hamlet which are not found in Much Ado About Nothing:
> perl6 bin/most-common-unique.pl data/Hamlet.txt data/Much_Ado_About_Nothing.txt
ham: 358
queen: 119
hamlet: 118
hor: 111
pol: 86
laer: 62
oph: 58
ros: 53
horatio: 48
clown: 47
December 13, 2012 15:50
December 12, 2012
Sometimes things go horribly wrong, and the only thing you can do is not to go on. Then you throw an exception.
But of course the story doesn’t end there. The caller (or the caller’s caller) must somehow deal with the exception. To do that in a sensible manner, the caller needs to have as much information as possible.
In Perl 6, exceptions should inherit from the type Exception, and by convention they go into the X:: namespace.
So for example if you write a HTTP client library, and you decide that an exception should be thrown when the server returns a status code starting with 4 or 5, you could declare your exception class as
class X::HTTP is Exception {
has $.request-method;
has $.url;
has $.status;
has $.error-string;
method message() {
"Error during $.request-method request"
~ " to $.url: $.status $.error-string";
}
}
And throw an exception as
die X::HTTP.new(
request-method => 'GET',
url => 'http://example.com/no-such-file',
status => 404,
error-string => 'Not found',
);
The error message then looks like this:
Error during GET request to
http://example.com/no-such-file: 404 Not found
(line wrapped for the benefit of small browser windows).
If the exception is not caught, the program aborts and prints the error message, as well as a backtrace.
There are two ways to catch exceptions. The simple Pokemon style “gotta catch ‘em all” method catches exception of any type with try:
my $result = try do-operation-that-might-die();
if ($!) {
note "There was an error: $!";
note "But I'm going to go on anyway";
}
Or you can selectively catch some exception types and handle only them, and rethrow all other exceptions to the caller:
my $result = do-operation-that-might-die();
CATCH {
when X::HTTP {
note "Got an HTTP error for URL $_.url()";
# do some proper error handling
}
# exceptions not of type X::HTTP are rethrown
}
Note that the CATCH block is inside the same scope as the one where the error might occur, so that by default you have access to all the interesting varibles from that scope, which makes it easy to generate better error messages.
Inside the CATCH block, the exception is available as $_, and is matched against all when blocks.
Even if you don’t need to selectively catch your exceptions, it still makes sense to declare specific classes, because that makes it very easy to write tests that checks for proper error reporting. You can check the type and the payload of the exceptions, without having to resort to checking the exact error message (which is always brittle).
But Perl 6 being Perl, it doesn’t force you to write your own exception types. If you pass a non-Exception objects to die(), it simply wraps them in an object of type X::AdHoc (which in turn inherits from Exception), and makes the argument available with the payload method:
sub I-am-fatal() {
die "Neat error message";
}
try I-am-fatal();
say $!; # Neat error message;
say $!.perl; # X::AdHoc.new(payload => "Neat error message")
To find out more about exception handling, you can read the documentation of class Exception and Backtrace.
December 12, 2012 05:12
December 11, 2012
Editors note: I, rurban, does know almost nothing about threads. Any errors are probably mine. I just tested them, fixed some deadlocks, added the numcpu code and merged the threads branch to master.
Parrot now supports fast and lightweight OS threads, based on Nat Tucks’s initial GSoC work together with Andrew “whiteknight” Whitworth on green threads and finally Stefan Seifert’s extension to true parallel OS threads as hybrid threads.
See http://wknight8111.blogspot.co.at/2010/08/gsoc-threads-chandons-results.html and http://niner.name/Hybrid_Threads_for_the_Parrot_VM.pdf
History
Parrot always supported “threads”, i.e. concurrency models over the last years, but we identified various problems with the particular designs and were continuously improving them. In our case without changing the API too much, as the pdd25 concurrency spec is pretty high level, describing the various models parrot should support, and also pretty low-level in describing the two PMC’s which export the threads API, the Task and the Scheduler classes.
Being born at a time when Perl 6 still looked much more similar to Perl 5 than it does nowadays, Parrot’s threading support initially was very close to Perl’s ithreads model. Previous attempts to change this into the more conventional model of data shared by default or implementing new technologies like STM “Software Transactional Memory” failed. For example Parrot has never supported running multiple threads and having garbage collection at the same time.
In the year 2005 development of faster Central Processing Units (CPUs) shifted from increased speed of a single core to adding more cores. Modern processors contain up to 12 cores with even mobile phones having up to four. To utilize a modern CPU’s power, code needs to be run in parallel. In UNIX (and thus Perl) tradition, this is accomplished using multiple processes being a good solution for many use cases. For many others like auto threading of hyper operators in Perl 6, the cost of process setup and communication would be prohibitively high except for very large data sets.
During the years of back and forth and failed attempts of adding threading support to Parrot, the Perl 6 specification evolved to a point where the largest parts of the language were covered and its features were implemented in the compilers. The lack of concurrency primitives in Parrot however prevents any progress in the area of concurrency support.
Summary
Green threads were used to simplify the implementation of a nearly lock free multithreading implementation.
Parrot supports now native Win32 threads and POSIX threads. Win32 alarm, sleep and premption is unified with POSIX, it is handled on a common timer thread.
Parrot creates at startup a thread pool of --numthreads threads, which defaults to the number of available CPU cores. Activating a new thread at runtime causes no run-time penalties, until the number of cores is utilized. When a user starts a new task, the scheduler first looks for an idle thread. If one can be found, the task is scheduled on the thread’s interpreter. If more tasks are started than the maximum number of threads, the tasks are distributed evenly among the running interpreters. This is effectively an implementation of the N:M threading model.
Green threads
Our GSOC student Nan “Chandor” Tuck worked in summer 2010 on green threads.
What I have working now is a pre-emptively scheduled green threads system for Parrot that allows programs to be written in a concurrent style. Individual green threads can do basic blocking file input without stopping other threads from running. These logical threads are accessed using the Task API that I described a couple weeks ago. This functionality makes Parrot similarly powerful at threading as the standard version of Ruby or Python: Threads can do pretty much everything except run at the same time. http://parrot.org/content/hybrid-threads-gsoc-project-results
What was missing from this green threads branch was true parallel execution in OS threads, one global_interpreter structure that is shared and protected by locks or other concurrent access rules and many local_interpreters that run simultaneously in separate OS threads.
OS threads
From Fall 2011 to Summer 2012 Stefan “nine” Seifert implemented true OS threads on top of green threads to finally allow true parallel execution of Tasks, to implement blocking IO, and to give perl6 some more advantages over perl5.
The lightweight “green” threads are used as messages in a system where reading shared variables is allowed but only the one owner thread may write to it. That’s why we call it hybrid threads.
Why is multithreading support so difficult to implement?
Low level programming languages like C provide only the bare necessities, leaving the responsibility for preventing data corruption and synchronization entirely to the user. A high-level language like Perl 6 on the other hand provides complex and compound data types, handles garbage collection and a very dynamic object system. Even seemingly simple things like a method call can become very complex. In a statically typed programming language the definition of a class is immutable. Thus, calling a method on an object contains just the steps of determining the object’s class, fetching the required method from this class and calling it. Calling the same method again can then even omit the first two steps since their results cannot change.
In a dynamic language, the object may change its class at runtime. The inheritance hierarchy of the class may be changed by adding or removing parent classes. Methods may be added to or removed from classes (or objects) at runtime and even the way how to find a method of a class may change. So a simple method call results in the following steps:
· determining the class of the object,
· determining the method resolution method of the class,
· finding the actual method to call,
· calling the method.
These steps have to be repeated for every method call, because the results may change any time. In a threaded environment, a thread running in parallel may change the underlying data and meta data in between those sequences and even between those steps. As a consequence, this meta data has to be protected from corruption introducing the need for locks in a performance critical area.
Many interpreters for dynamic languages like Python or Ruby handle this problem by using a global interpreter lock (GIL) to effectively serialize all operations. This is a proven and reliable way but leaves much of the hardware’s potential unused.
Java
In Java, the user is responsible for preventing concurrency issues. The language provides synchronization primitives like mutexes, but the interpreter (the Java Virtual Machine, JVM) does not protect the consistency of the provided data structures. The class library provides the user with high-level data structures explicitly designed for multithreaded scenarios.
Java version 1.1 used green threads to support multithreaded execution of Java programs. Green threads are threads simulated by the virtual machine (VM) but unable to use more than one CPU core for processing. Version 1.2 introduced native Operating System (OS) threading support which since has become the standard way to do multithreading in Java.
Python
The CPython implementation of the Python runtime uses a Global Interpreter Lock (GIL) to protect its internal consistency. This is a single lock taken whenever the interpreter executes Python bytecode. Because of this lock, only one thread can execute bytecode at any time so all built-in types and the object model are implicitly type safe. The drawback is that Python code cannot benefit from having multiple CPU cores available. However I/O operations and calls to external libraries are executed without holding the GIL, so in applications with multiple I/O bounded threads, there may still be a performance benefit from using multithreading.
To run Python code in parallel, multiple processes have to be used. The multiprocessing module provides support for spawning processes exposed through an API similar to the threading module. Since processes may not directly access other processes’ memory, the multiprocessing module provides several means of communication between processes: Queues, Pipes and shared memory support.
Parrot
Much of Parrot’s previous threading related code has been removed to clean up the code and improve performance. Since the existing threading support was known to be unreliable and seriously flawed, this was no trade off. The final parts were removed by the merging of the kill_threads branch on September, 21st 2011.
In 2010, Nat Tuck began working on a green_threads branch during his Google Summer of Code internship. The feature got prototyped using pure PIR and then implemented in Parrot’s core. He got it to work in simple cases and started to work on OS thread support but the internship ended before the code was ready to be merged into the master branch. The code lay dormant until the work on hybrid threads in the threads branch started in 2011.
In Parrot, green threads are called Tasks. Each task is assigned a fixed amount of execution time. After this time is up a timer callback sets a flag which is checked at execution of every branch operation. Since the interpreter’s state is well defined at this point, its internal consistency is guaranteed. The same holds for the GC. Since task preemption is only done while executing user-level code, the GC can do its work undisturbed and without the need for measures like locking. Since user-level code is allowed to disable the scheduler, it can be guaranteed to run undisturbed through critical sections.
The scheduler is implemented as a PMC type. This allows the user to subclass this PMC thus allowing fine-grained control over the scheduling policy. Features, a user could add this way would be for example giving different priorities to tasks or implementing the possibility to suspend and resume a task.
Shared data
Cross-thread writes to shared variables may endanger the internal consistency of the interpreter. Traditionally, the solution to this problem is the use of locks of varying granularity. Fine-grained locking allows code to run in parallel but taking and releasing locks costs performance. It not only increases the instruction count and memory accesses but it also forces the CPU cores to coordinate and thus communicate. Even a seemingly simple operation like an atomic increment can take two orders of magnitude longer than a normal increment. While the gain through being able to utilize multiple CPU cores may offset this cost, it is still impacting the common case of having only a single thread running.
Too coarse locking on the other hand would reduce scalability and the performance gains through parallel execution by having threads wait for extended periods for locks to become available. In the extreme case of having a global interpreter lock it would effectively serialize all computations costing much of the benefits of using threads in the first place.
The other problem with locking is the possibility of introducing deadlocks. For example, two functions F1 and F2 both use two resources A and B protected by locks. If F1 first locks A and then tries to lock B while F2 has already locked B and is now trying to lock A, the program would come to a halt. Both functions would be left waiting for the other to unlock the resource which will never happen. With fine-grained locking, the possibilities for such bugs grow quickly. At the same time, it is easy to miss a case where a lock would be appropriate leading to difficult to diagnose corruption bugs.
The solution for these problems implemented in hybrid threads is to sidestep them altogether by disallowing write access to shared variables. The programmer (or in most cases the compiler) is obliged to declare a list of all shared variables before a newly created task is started. The interpreter would then create proxy objects for these variables which the task can use to access the data. These proxies contain references to the original objects. They use these references to forward all reading vtable functions to the originals. Write access on the other hand would lead to a runtime error.
In other words, all data is owned by the thread creating it and only the owner may write to it. Other threads have only read access.
For threads to be able to communicate with their creators and other threads, they still need to write to shared variables. This is where green threads come into play. Since green threads are light weight, it is feasible for a thread to create a task just for updating a variable. This task is scheduled on the interpreter owning this variable. To reduce latency, the task is flagged to run immediately. The data-owning interpreter will preempt the currently running task and process the new write task. Put another way, the data-owning interpreter is told what to write to its variables, so other threads don’t have to.
Proxies
Proxies are the arbiters between threads. They are the only means for a thread to access another thread’s data and are implemented by the Proxy PMC type.
Proxy has default implementations for all functions, writing functions raise a cant_do_write_method runtime exception. If a method returns a PMC from the target’s interp, another proxy object has to be created and wrapped around it so it can be safely returned to the caller.
Sub
The Sub PMC represents executable subroutines. A Sub does not only contain the code to execute but also the context in which to execute the code such as visible globals and namespaces. If a proxy to such a Sub were created and invoke called on it, the code would access this context directly since it belongs to the same interp as the proxied Sub itself. Thus, an operation like get_global fetches a global from an unproxied namespace and an unproxied global is be put into the target register. Since this is happening while running invoke on the original Sub, Proxy cannot intercept the call and create a Proxy for the result.
This is the reason why Parrot_thread_create_proxy does not create a Proxy for a Sub but uses Parrot_thread_create_local_sub to create a copy on the thread’s interp with proxies for all PMC attributes.
Writing to shared variables
To write to shared variables, a thread creates a task and schedules it on the data owning interpreter. An example task looks like this:
.sub write_to_variable
.param pmc variable
variable = 1
.end
This is a subroutine with just one parameter. The variable passed as this parameter is the one the task should write to. In this case the constant value 1 would be written to the variable. In PIR, an assignment to a PMC gets translated to a method call. In this case, the set_integer_native is called changing the variable’s value. Since PMCs are passed by reference, it is the original variable which gets written to.
Code to create the task looks like:
1 write_task = new ['Task']
2 setattribute write_task, 'code', write_to_variable
3 setattribute write_task, 'data', shared_variable
4 interp.'schedule_proxied'(write_task, shared_variable)
Line 1 creates a new task object. The example subroutine is used for the task’s code attribute. shared_variable is used for data. At this point, shared_variable is actually the proxy object created for the shared integer PMC. The interpreter object contains a schedule_proxied method which is used to schedule the write_task on the thread owning the original variable.
schedule_proxied uses Parrot_thread_create_local_task which in this case detects that the data given as parameter for the task’s code is actually a proxy already and unwraps the proxied object. Parrot_cx_schedule_immediate is then used to make the data owning interpreter execute the task as soon as possible.
To protect a critical section, preemption can be disabled so the critical section runs uninterrupted:
1 .sub swap_variables
2 .param pmc a, b
3 .local temp
4 disable_preemption
5 temp = a
6 a = b
7 b = temp
8 enable_preemption
9 .end
wait
Using tasks to write to shared variables makes such actions inherently asynchronous. This is not always what is needed by the implemented algorithm. For example, when the shared variable is a lock, processing should continue as soon as it’s acquired. The wait operation is used to wait for a task’s completion. The waiting task is added to the waited for task’s waiters list and preempted immediately. When a task finishes, all the tasks in the waiters list are scheduled again for execution. Since for each task a local copy is created on the target thread, the running task not only checks its own waiters list but also its partner’s.
If a task on the main thread was waiting for a task on another thread to finish and no other tasks are in the scheduler’s queue on the main thread, the main thread exits if no alarms are pending. To prevent this unintended exit, all tasks are added to the scheduler’s foreign_tasks list when they are scheduled on other threads. To end the program with other threads still running, an explicit exit operation has to be used.
Benchmarks
Preliminary benchmarks have shown Parrot’s performance to be within an order of magnitude of that of an optimized implementation in Perl 5.
Since Parrot does not yet offer the user any synchronization primitives, locks had to be implemented using a shared variable which is written to only by the main thread. Replacing this primitive method with a native semaphore implementation would probably reduce runtime to a small fraction.
Runtime comparison for matrix multiplication
singlethreaded computation multithreaded computation
1. run 28.522 s 19.530 s 17.543 s 8.478 s
2. run 28.427 s 19.463 s 17.320 s 8.283 s
3. run 28.200 s 19.235 s 17.489 s 8.473 s
average 28.383 s 19.409 s 17.451 s 8.411 s
This test implements matrix multiplication using four threads. For simplicity the second matrix only has one column. The program is written in the Winxed programming language. Winxed is a low-level language with Javascript like syntax and the possibility to include sections of PIR code verbatim making it possible to try experimental opcodes while writing more readable and concise code than with PIR alone. The complete source code is available in examples/threads/matrix_part.winxed
The program consists of the parts initialization, computation and verification. Computation is parallelized using four tasks each calculating one fourth of the result vector. Runtime is compared to a simple singlethreaded implementation. Run times were measured using the time command and are recorded in the above table.
As can be seen, the multithreaded implementation gives an average speedup of 2.31 for the computation and 1.61 in total.
Runtime comparison for Mandelbrot set calculation
singlethreaded 1 thread 2 threads 4 threads 8 threads
1. run 89.931 s 89.978 s 45.813 s 24.028 s 17.445 s
2. run 89.707 s 89.871 s 45.906 s 24.048 s 17.695 s
3. run 90.318 s 89.839 s 45.951 s 24.049 s 17.573 s
average 89.985 s 89.896 s 45.890 s 24.042 s 17.571 s
speedup 1.000 1.001 1.959 3.739 5.116
The complete source code is available in examples/pir/mandel.pir
Calculating an image of the Mandelbrot set is a common benchmark for multithreading implementations since calculations of points are independent of each other and are thus easily parallelizable. A simple implementation of the escape time algorithm written in Winxed has been used to determine scalability properties of the threading implementation. The image is split into lines which are calculated alternatedly by a configured number of tasks. Run times were measured using the time command on an Intel Core i7 3770K processor with 16 GiB RAM running openSUSE 12.1 and are recorded in the Mandelbrot table. As can be seen, the implementation scales nearly linearly up to four threads reflecting the CPU’s four physical cores. Using eight threads, the speedup is only 1.368 compared to four threads but this seems to be more a limitation of the hardware than the implementation.
Questions
On IRC and on the mailing list some detailed questions have been asked.
See here:
http://lists.parrot.org/pipermail/parrot-dev/2012-December/007295.html
December 11, 2012 00:08
December 09, 2012
Perl 6 regular expressions prefer to match the longest alternative when possible.
say "food and drink" ~~ / foo | food /; # food
This is in contrast to Perl 5, which would prefer the first alternative above, and produce the match “foo”.
You can still get the first-alternative behavior if you want; it’s tucked away in the slightly longer alternation operator ||:
say "food and drink" ~~ / foo || food /; # foo
…And that’s it! That’s Longest Token Matching. ☺ Short post.
“Huh, wait!” I hear you exclaim, in a desperate attempt to make the daily Perl 6 Advent goodness last a bit longer. “Why is Longest Token Matching such a big deal? Who would ever be so obsessed with long tokens?”
I’m glad you asked. As it turns out, Longest Token Matching (or LTM for short) plays very well with our intuition about how things should be parsed. If you’re creating a language, you want people to be able to declare a variable forest_density without the mention of this variable clashing with the syntax of for loops. LTM will see to that.
I like “strange consistencies” — when distal parts of a language design turn out to have commonalities that make the language feel more uniform. There is that kind of consistency here, between classes and grammars. Perl 6 basically exploits that consistency to the max. Let me briefly map out what I mean.
We’re all used to writing classes at this point. From a birds-eye view, they look like this:
class {
method
method
method
}
Grammars have a suspiciously similar structure:
grammar {
rule
rule
rule
}
(The keywords are actually regex, token and rule, but when we talk about them as a group, we just call them “rules”.)
We’re also used to being able to derive classes into subclasses (class B is A), and add or override methods in a way which produces a nice mix of old and new behavior. Perl 6 provides multi methods which even allow you to add new methods of the same name, and the old ones won’t be overridden, they’ll just all try to match alongside the new methods. The dispatch is handled by a (usually autogenerated) proto method that dispatches to all eligible candidates.
What does all this have to do with grammars and rules? Well, it turns out that first off, you can derive new grammars from old ones. It works the same as deriving classes. (In fact, under the hood it’s exactly the same mechanism. Grammars are classes with a different metaclass object.) New rules will override old rules just like you’d expect with methods.
S05 has a cute example with parsing of letters, and deriving the grammar to parse formal letters:
grammar Letter {
rule text { }
rule greet { [Hi|Hey|Yo] $=(\S+?) , $$}
rule body { +? } # note: backtracks forwards via +?
rule close { Later dude, $=(.+) }
}
grammar FormalLetter is Letter {
rule greet { Dear $=(\S+?) , $$}
rule close { Yours sincerely, $=(.+) }
}
The derived FormalLetter overrides greet and close, but not body.
But what about all the goodness with multi methods? Could we define some kind of “proto rule” that would allow us to have several rules in a grammar with the same name but different bodies? For example, we might want to parse a language with a rule term, but there are many different terms: strings, numbers… and maybe the numbers can be decimal or binary or octal or hexadecimal…
Perl 6 grammars can contain a proto rule, and then you can define and redefine a rule with the same name as many times as you want. And now we’re back full circle with the / foo | food / alternation from the start of the article. All those rules you write with the same name compile down to one big alternation. Not only that — rules which call other rules, some of them possibly proto rules, all of that will be “flattened” out into one big LTM alternation. In practice that means that all the possible things a term can be are tried out all at once, on equal footing. Neither alternative wins because you happened to define it before the others. An alternative wins because it is the longest.
The strange consistency resides in the fact that in the call-a-method side of things, the most specific method wins, and “most specific” has to with signature narrowness. The better the types in the signature describe the arguments coming in, the more specific the method.
In the parse-with-a-rule side of things, the most specific rule wins, but here “most specific” has to do with parse success. The better the rule can describe what comes next in the text, the more specific the rule.
And that’s strangely consistent, because on the surface methods and rules look like quite different beasts.
We really believe we have something going with this whole principle of deriving a grammar and getting a new language. LTM is right at the center of that because it allows new rules and old to intermix in a fair and predictable way. It’s a kind of meritocracy: rules win not based on whether they’re young or old, but based on whether they are able to parse the text well.
In fact, the Perl 6 compiler itself works this way. It parses your program using a Perl 6 grammar, and that grammar is derivable… whenever you declare a new operator in your program, a new grammar is derived for you. The parsing of your operator is added as a new rule in the new grammar, and the new grammar is given the task of parsing the rest of your program. Your new operator will win against similar but shorter ones, and lose against similar but longer ones.
December 09, 2012 00:01
December 08, 2012
Perl 6 is not just the language. While without modules it can do more than Perl 5, modules can make life easier. About two years ago neutro was discussed on this blog. I’m not going to talk about it, as it’s deprecated today.
Today, the standard way of installing modules is the panda utility. If you’re using Rakudo Star, you should have it already installed (try the panda command in console to check it). After running it and waiting a few seconds, you should see help for the panda utility.
$ panda
Usage:
panda [--notests] [--nodeps] install [ ...] -- Install the specified modules
panda [--installed] [--verbose] list -- List all available modules
panda update -- Update the module database
panda info [ ...] -- Display information about specified modules
panda search -- Search the name/description
As you can see, it doesn’t have many options (it’s actually similar to RubyGems or cpanminus in its simplicity). You can see the current list of modules at Perl 6 Modules page. Let’s say you would want to parse an INI file. First, you can find module for it using the search command.
$ panda search INI
JSON::Tiny * A minimal JSON (de)serializer
Config::INI * .ini file parser and writer module for
Perl 6
MiniDBI * a subset of Perl 5 DBI ported to Perl 6
to use while experts build the Real Deal
Class::Utils 0.1.0 Small utilities to help with defining
classes
Config::INI is module you want. Other modules were found because my query wasn’t specific enough and found “ini” in other words (minimal, MiniDBI and defining). Config::INI isn’t part of Rakudo Star, so you have to install it.
Panda installs modules globally when you writing access to the installation directory, locally otherwise. Because of that you can use panda even when Perl 6 is installed globally without installing modules like local::lib, like you have to in Perl 5.
$ panda install Config::INI
==> Fetching Config::INI
==> Building Config::INI
Compiling lib/Config/INI.pm
Compiling lib/Config/INI/Writer.pm
==> Testing Config::INI
t/00-load.t .... ok
t/01-parser.t .. ok
t/02-writer.t .. ok
All tests successful.
Files=3, Tests=55, 3 wallclock secs ( 0.04 usr 0.00 sys + 2.38 cusr 0.14 csys = 2.56 CPU)
Result: PASS
==> Installing Config::INI
==> Succesfully installed Config::INI
After the module has been installed, you can update it as easily – by installing it. Currently panda cannot automatically upgrade modules, but after a module has been updated (you can watch repositories on GitHub to know when it happens – every module is available on GitHub), you can easily upgrade it by reinstalling the module.
When a module was installed, you can check if it works by trying to use it. This is a sample script that can be used to convert INI file into a Perl 6 data structure.
#!/usr/bin/env perl6
use Config::INI;
multi sub MAIN($file) {
say '# your INI file as seen by Perl 6';
say Config::INI::parse_file($file).perl;
}
December 08, 2012 12:49
To begin with, I’ve made a post for Perl 6 Advent Calendar. If you were wondering why it was late, I will tell you - it’s actually early. Somebody else has claimed article for eighth day and forgot to upload it, so somebody had to put replacement article. I’ve moved my article from 12th day to 8th day, just so Perl 6 Advent Calendar could be continued. Moritz, to protect against that, put the just-in-case article about exceptions (it’s exceptional), if that would happen again.
Enough talking about Perl 6 Advent Calendar, as obviously this blog post isn’t about it. Every week, I describe changes in Perl 6 and its implementations. This is yet another article about changes.
Jonathan Worthington lately was working on improving syntax errors messages in Rakudo Perl. Now they are better and more accurate, just like they’re already in STD.pm6 or Niecza.
New features
Rakudo Perl
- Syntax errors now provide context using ⏏ symbol, just like STD.pm6 does.
- Rakudo is now more accurate about error locations and doesn’t inform about errors on line 1 (like missing semicolon) that actually are on line 777.
- Syntax errors now have colors.
Niecza Perl
- A
.join method isn’t O(N4) anymore.
- A
~ infix operator is faster.
- Numeric ranges are faster.
December 08, 2012 08:00
December 07, 2012
parrot MIME::Base64 FixedIntegerArray: index out of bounds!
Ronaldxs created the following parrot ticket #813 4 months ago:
“Was playing with p6 MIME::Base64 and utf8 sampler page when I came across this. It seems that the parrot MIME Base64 library can’t handle some UTF-8 characters as demonstrated below.”
.sub go :main
load_bytecode 'MIME/Base64.pbc'
.local pmc enc_sub
enc_sub = get_global [ "MIME"; "Base64" ], 'encode_base64'
.local string result_encode
result_encode = enc_sub(utf8:"\x{203e}")
say result_encode
.end
FixedIntegerArray: index out of bounds!
current instr.: 'parrot;MIME;Base64;encode_base64'
pc 163 (runtime/parrot/library/MIME/Base64.pir:147)
called from Sub 'go' pc 11 (die_utf8_base64.pir:8)
This was interesting, because parrot strings store the encoding information in the string. The user does not need to store the string encoding information somewhere else as in perl5, nor have to do educated guesses about the encoding. parrot supports ascii, latin1, binary, utf-8, ucs-2, utf-16 and ucs-4 string encodings natively.
So we thought we the hell cannot parrot handle simple utf-8 encoded strings?
As it turned out, the parrot implementation of MIME::Base64, which can be shared to all languages which use parrot as VM, stored the character codepoints for each character as array of integers. On multibyte encodings such as UTF-8 this leads to different data held in memory than a normal multibyte string which is encoded as the byte buffer and the additional encoding information.
Internal string representations
For example an overview of different internal string representations for the utf-8 string “\x{203e}”:
perl5 strings:
len=3, utf-8 flag, "\342\200\276" buf=[e2 80 be]
parrot strings:
len=1, bufused=3, encoding=utf-8, buf=[e2 80 be]
The Unicode tables:
U+203E ‾ e2 80 be OVERLINE
gdb perl5
Let’s check it out:
$ gdb --args perl -e'print "\x{203e}"'
(gdb) start
(gdb) b Perl_pp_print
(gdb) c
(gdb) n
.. until if (!do_print(*MARK, fp))
(gdb) p **MARK
$1 = {sv_any = 0x404280, sv_refcnt = 1, sv_flags = 671106052, sv_u = {
svu_pv = 0x426dd0 "‾", svu_iv = 4353488, svu_uv = 4353488,
svu_rv = 0x426dd0, svu_array = 0x426dd0, svu_hash = 0x426dd0,
svu_gp = 0x426dd0, svu_fp = 0x426dd0}, ...}
(gdb) p Perl_sv_dump(*MARK)
ALLOCATED at -e:1 for stringify (parent 0x0); serial 301
SV = PV(0x404280) at 0x4239a8
REFCNT = 1
FLAGS = (POK,READONLY,pPOK,UTF8)
PV = 0x426dd0 "\342\200\276" [UTF8 "\x{203e}"]
CUR = 3
LEN = 16
$2 = void
(gdb) x/3x 0x426dd0
0x426dd0: 0xe2 0x80 0xbe
We see that perl5 does store the utf-8 flag, but not the length of the string, the utf8 length (=1), only the length of the buffer (=3).
Any other multi-byte encoded string, such as UCS-2 is stored differently. We suppose as utf-8.
We are already in the debugger, so let’s try the different cmdline argument.
(gdb) run -e'use Encode; print encode("UCS-2", "\x{203e}")'
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Breakpoint 2, Perl_pp_print () at pp_hot.c:712
712 dVAR; dSP; dMARK; dORIGMARK;
(gdb) p **MARK
$3 = {sv_any = 0x404b30, sv_refcnt = 1, sv_flags = 541700, sv_u = {
svu_pv = 0x563a50 " >", svu_iv = 5651024, svu_uv = 5651024,
svu_rv = 0x563a50, svu_array = 0x563a50, svu_hash = 0x563a50, svu_gp = 0x563a50,
svu_fp = 0x563a50}, ...}
(gdb) p Perl_sv_dump(*MARK)
ALLOCATED at -e:1 by return (parent 0x0); serial 9579
SV = PV(0x404b30) at 0x556fb8
REFCNT = 1
FLAGS = (TEMP,POK,pPOK)
PV = 0x563a50 " >"
CUR = 2
LEN = 16
$4 = void
(gdb) x/2x 0x563a50
0x563a50: 0x20 0x3e
But we don’t see the UTF8 flag in encode(“UCS-2″, “\x{203e}”), just the simple ascii string ” >”, which is the UCS-2 representation of [20 3e].
Because ” >” is perfectly representable as non-utf8 ASCII string.
UCS-2 is much much nicer than UTF-8, is has a fixed size, it is readable, Windows uses it, but it cannot represent all Unicode characters.
Encode::Unicode contains this nice cheatsheet:
Quick Reference
Decodes from ord(N) Encodes chr(N) to...
octet/char BOM S.P d800-dfff ord > 0xffff \x{1abcd} ==
---------------+-----------------+------------------------------
UCS-2BE 2 N N is bogus Not Available
UCS-2LE 2 N N bogus Not Available
UTF-16 2/4 Y Y is S.P S.P BE/LE
UTF-16BE 2/4 N Y S.P S.P 0xd82a,0xdfcd
UTF-16LE 2/4 N Y S.P S.P 0x2ad8,0xcddf
UTF-32 4 Y - is bogus As is BE/LE
UTF-32BE 4 N - bogus As is 0x0001abcd
UTF-32LE 4 N - bogus As is 0xcdab0100
UTF-8 1-4 - - bogus >= 4 octets \xf0\x9a\af\8d
---------------+-----------------+------------------------------
gdb parrot
Back to parrot:
If you debug parrot with gdb you get a gdb pretty-printer thanks to Nolan Lum, which displays the string and encoding information automatically.
In perl5 you have to call Perl_sv_dump with or without the my_perl as first argument, if threaded or not. With a threaded perl, e.g. on Windows you’d need to call p Perl_sv_dump(my_perl, *MARK).
In parrot you just ask for the value and the formatting is done with a gdb pretty-printer plugin.
The string length is called strlen (of the encoded string), the buffer size is called bufused.
Even in a backtrace the string arguments are displayed abbrevated like this:
#3 0x00007ffff7c29fc4 in utf8_iter_get_and_advance (interp=0x412050, str="utf8:� [1/2]",
i=0x7fffffffdd00) at src/string/encoding/utf8.c:551
#4 0x00007ffff7a440f6 in Parrot_str_escape_truncate (interp=0x412050, src="utf8:� [1/2]",
limit=20) at src/string/api.c:2492
#5 0x00007ffff7b02fb3 in trace_op_dump (interp=0x412050, code_start=0x63a1c0, pc=0x63b688)
at src/runcore/trace.c:450
[1/2] means strlen=1 bufused=2
Each non-ascii or non latin-1 encoded string is printed with the encoding prefix.
Internally the encoding is of course a index or pointer in the table of supported encodings.
You can set a breakpoint to utf8_iter_get_and_advance and watch the strings.
(gdb) r t/library/mime_base64u.t
Breakpoint 1, utf8_iter_get_and_advance (interp=0x412050, str="utf8:\\x{00c7} [8/8]",
i=0x7fffffffcd40) at src/string/encoding/utf8.c:544
(gdb) p str
$1 = "utf8:\\x{00c7} [8/8]"
(gdb) p str->bufused
$3 = 8
(gdb) p str->strlen
$4 = 8
(gdb) p str->strstart
$5 = 0x5102d7 "\\x{00c7}"
This is escaped. Let’s advance to a more interesting utf8 string in this test, i.e. until str=”utf8:Ā [1/2]“
You get the members of a struct with tab-completion, i.e. press <TAB> after p str->
(gdb) p str->
_buflen _bufstart bufused encoding flags hashval strlen strstart
(gdb) p str->strlen
$9 = 8
(gdb) dis 1
(gdb) b utf8_iter_get_and_advance if str->strlen == 1
(gdb) c
Breakpoint 2, utf8_iter_get_and_advance (interp=0x412050, str="utf8:Ā [1/2]",
i=0x7fffffffcd10) at src/string/encoding/utf8.c:544
544 ASSERT_ARGS(utf8_iter_get_and_advance)
(gdb) p str->strlen
$10 = 1
(gdb) p str->strstart
$11 = 0x7ffff7faeb58 "Ā"
(gdb) x/2x str->strstart
0x7ffff7faeb58: 0xc4 0x80
(gdb) p str->encoding
$12 = (const struct _str_vtable *) 0x7ffff7d882e0
(gdb) p *str->encoding
$13 = {num = 3, name = 0x7ffff7ce333f "utf8", name_str = "utf8", bytes_per_unit = 1,
max_bytes_per_codepoint = 4, to_encoding = 0x7ffff7c292b0 <utf8_to_encoding>, chr =
0x7ffff7c275c0 <unicode_chr>, equal = 0x7ffff7c252e0 <encoding_equal>, compare =
0x7ffff7c254e0 <encoding_compare>, index = 0x7ffff7c25690 <encoding_index>, rindex
= 0x7ffff7c257a0 <encoding_rindex>, hash = 0x7ffff7c25a20 <encoding_hash>, scan =
0x7ffff7c29380 <utf8_scan>, partial_scan = 0x7ffff7c29460 <utf8_partial_scan>, ord
= 0x7ffff7c297e0 <utf8_ord>, substr = 0x7ffff7c25de0 <encoding_substr>, is_cclass =
0x7ffff7c26000 <encoding_is_cclass>, find_cclass =
0x7ffff7c260e0 <encoding_find_cclass>, find_not_cclass =
0x7ffff7c26220 <encoding_find_not_cclass>, get_graphemes =
0x7ffff7c263d0 <encoding_get_graphemes>, compose =
0x7ffff7c27680 <unicode_compose>, decompose = 0x7ffff7c26450 <encoding_decompose>,
upcase = 0x7ffff7c27b20 <unicode_upcase>, downcase =
0x7ffff7c27be0 <unicode_downcase>, titlecase = 0x7ffff7c27ca0 <unicode_titlecase>,
upcase_first = 0x7ffff7c27d60 <unicode_upcase_first>, downcase_first =
0x7ffff7c27dc0 <unicode_downcase_first>, titlecase_first =
0x7ffff7c27e20 <unicode_titlecase_first>, iter_get =
0x7ffff7c29c40 <utf8_iter_get>, iter_skip = 0x7ffff7c29d60 <utf8_iter_skip>,
iter_get_and_advance = 0x7ffff7c29eb0 <utf8_iter_get_and_advance>,
iter_set_and_advance = 0x7ffff7c29fd0 <utf8_iter_set_and_advance>}
encode_base64(str)
$ perl -MMIME::Base64 -lE'$x="20e3";$s="\x{20e3}";
printf "0x%s\t%s=> %s",$x,$s,encode_base64($s)'
Wide character in subroutine entry at -e line 1.
Oops, I’m clearly a unicode perl5 newbie. Does my term not understand utf-8?
$ echo $TERM
xterm
No, it should. encode_base64 does not understand unicode.
perldoc MIME::Base64
“The base64 encoding is only defined for single-byte characters. Use the Encode module to select the byte encoding you want.”
Oh my! But it is just perl5. It just works on byte buffers, not on strings.
perl5 strings can be utf8 and non-utf8. Why on earth an utf8 encoded string is disallowed and only byte buffers of unknown encodings are allowed goes beyond my understanding, but what can you do. Nothing. base64 is a binary only protocol, based on byte buffers. So we decode it manually to byte buffers. The Encode API for decoding is called encode.
$ perl -MMIME::Base64 -MEncode -lE'$x="20e3";$s="\x{20e3}";
printf "0x%s\t%s=> %s",$x,$s,encode_base64(encode('utf8',$s))'
Wide character in printf at -e line 1.
0x20e3 => 4oOj
This is now the term warning I know. We need -C
$ perldoc perluniintro
$ perl -C -MMIME::Base64 -MEncode -lE'$x="20e3";$s="\x{20e3}";
printf "0x%s\t%s=> %s",$x,$s,encode_base64(encode('utf8',$s))'
0x20e3 => 4oOj
Over to rakudo/perl6 and parrot:
$ cat >m.pir << EOP
.sub main :main
load_bytecode 'MIME/Base64.pbc'
$P1 = get_global [ "MIME"; "Base64" ], 'encode_base64'
$S1 = utf8:"\x{203e}"
$S2 = $P1(s1)
say $S1
say $S2
.end
EOP
$ parrot m.pir
FixedIntegerArray: index out of bounds!
current instr.: 'parrot;MIME;Base64;encode_base64'
pc 163 (runtime/parrot/library/MIME/Base64.pir:147)
The perl6 test, using the parrot library, from https://github.com/ronaldxs/perl6-Enc-MIME-Base64/
$ git clone git://github.com/ronaldxs/perl6-Enc-MIME-Base64.git
Cloning into 'perl6-Enc-MIME-Base64'...
$ PERL6LIB=perl6-Enc-MIME-Base64/lib perl6 <<EOP
use Enc::MIME::Base64;
say encode_base64_str("\x203e");
EOP
> use Enc::MIME::Base64;
Nil
> say encode_base64_str("\x203e");
FixedIntegerArray: index out of bounds!
...
The pure perl6 workaround:
$ PERL6LIB=perl6-Enc-MIME-Base64/lib perl6 <<EOP
use PP::Enc::MIME::Base64;
say encode_base64_str("\x203e");
EOP
> use PP::Enc::MIME::Base64;
Nil
> say encode_base64_str("\x203e");
4oC+
Wait. perl6 creates a different enoding than perl5?
What about coreutils base64 command.
$ echo -n "‾" > m.raw
$ od -x m.raw
0000000 80e2 00be
0000003
$ ls -al m.raw
-rw-r--r-- 1 rurban rurban 3 Dec 6 10:23 m.raw
$ base64 m.raw
4oC+
[80e2 00be] is the little-endian version of [e2 80 be], 3 bytes, flipped.
Ok, at least base64 agrees with perl6, and I must have made some encoding mistake with perl5.
Back to debugging our parrot problem:
parrot unlike perl6 has no debugger yet. So we have to use gdb, and we need to know in which function the error occured. We use the parrot -t trace flag, which is like the perl5 debugging -Dt flag, but it is always enabled, even in optimized builds.
$ parrot --help
...
-t --trace [flags]
--help-debug
...
$ parrot --help-debug
...
--trace -t [Flags] ...
0001 opcodes
0002 find_method
0004 function calls
$ parrot -t7 m.pir
...
009f band I9, I2, 63 I9=0 I2=0
00a3 set I10, P0[I5] I10=0 P0=FixedIntegerArray=PMC(0xff7638) I5=[2063]
016c get_results PC2 (1), P2 PC2=FixedIntegerArray=PMC(0xedd178) P2=PMCNULL
016f finalize P2 P2=Exception=PMC(0x16ed498)
0171 pop_eh
lots of error handling
...
0248 callmethodcc P0, "print" P0=FileHandle=PMC(0xedcca0)
FixedIntegerArray: index out of bounds!
We finally see the problem, which matches the run-time error.
00a3 set I10, P0[I5] I10=0 P0=FixedIntegerArray=PMC(0xff7638) I5=[2063]
We want to set I10 to the I5=2063′th element in the FixedIntegerArray P0, and the array is not big enough.
After several hours of analyzing I came to the conclusion that the parrot library MIME::Base64 was wrong by using ord of every character in the string. It should use a bytebuffer instead.
Which was fixed with commit 3a48e6. ord can return integers > 255, but base64 can only handle chars < 255.
The fixed parrot library was now correct:
$ parrot m.pir
‾
4oC+
But then the tests started failing. I spent several weeks trying to understand why the parrot testsuite was wrong with the mime_base64 tests, the testdata came from perl5. I came up with different implementation hacks which would match the testsuite, but finally had to bite the bullet, changing the tests to match the implementation.
And I had to special case the tests for big-endian, as base64 is endian agnostic. You cannot decode a base64 encoded powerpc file on an intel machine, when you use multi-byte characters. And utf-8 is even more multi-byte than ucs-2. I had to accept the fact the big-endian will return a different encoding. Before the results were the same. The tests were written to return the same encoding on little and big-endian.
Summary
The first reason why I wrote this blog post was to show how to debug into crazy problems like this, when you are not sure if the core implementation, the library, the spec or the tests are wrong. It turned out, that the library and the tests were wrong.
You saw how easily you could use gdb to debug into such problems, as soon as you find out a proper breakpoint.
The internal string representations looked like this:
MIME::Base64 internally:
len=1, encoding=utf-8, buf=[3e20]
and inside the parrot imcc compiler the SREG
len=8, buf="utf-8:\"\x{203e}\""
parrot is a register based runtime, and a SREG is the string representation of the register value. Unfortunately a SREG cannot hold the encoding info yet, so we prefix the encoding in the string, and unquote it back. This is not the reason why parrot is still slower than the perl5 VM. I benchmarked it. parrot still uses too much sprintf’s internally and the encoding quote/unquoting counts only for a 4th of the time of the sprintf gyrations.
And parrot function calls are awfully slow and de-optimized.
The second reason is to explain the new decode_base64() API, which only parrot – and therefore all parrot based languages like rakudo – now have got.
decode_base64(str, ?:encoding)
“Decode a base64 string by calling the decode_base64() function.
This function takes as first argument the string to decode, as optional second argument the encoding string for the decoded data.
It returns the decoded data.
Any character not part of the 65-character base64 subset is silently ignored.
Characters occurring after a ‘=’ padding character are never decoded.”
So decode_base64 got now a second optional encoding argument. The src string for encode_base64 can be any encoding and is automatically decoded to a bytebuffer. You can easily encode an image or unicode string without any trouble, and for the decoder you can define the wanted encoding beforehand. The result can be the encoding binary or utf-8 or any encoding you prefer, no need for additional decoding of the result. The default encoding of the decoded string is either ascii, latin-1 or utf-8. parrot will upgrade the encoding automatically.
You can compare the new examples of pir against the perl5 version:
parrot:
.sub main :main
load_bytecode 'MIME/Base64.pbc'
.local pmc enc_sub
enc_sub = get_global [ "MIME"; "Base64" ], 'encode_base64'
.local string result_encode
# GH 814
result_encode = enc_sub(utf8:"\x{a2}")
say "encode: utf8:\"\\x{a2}\""
say "expected: wqI="
print "result: "
say result_encode
# GH 813
result_encode = enc_sub(utf8:"\x{203e}")
say "encode: utf8:\"\\x{203e}\""
say "expected: 4oC+"
print "result: "
say result_encode
.end
perl5:
use MIME::Base64 qw(encode_base64 decode_base64);
use Encode qw(encode);
my $encoded = encode_base64(encode("UTF-8", "\x{a2}"));
print "encode: utf-8:\"\\x{a2}\" - ", encode("UTF-8", "\x{a2}"), "\n";
print "expected: wqI=\n";
print "result: $encoded\n";
print "decode: ",decode_base64("wqI="),"\n\n"; # 302 242
my $encoded = encode_base64(encode("UTF-8", "\x{203e}"));
print "encode: utf-8:\"\\x{203e}\" -> ",encode("UTF-8", "\x{203e}"),"\n";
print "expected: 4oC+\n";
print "result: $encoded\n"; # 342 200 276
print "decode: ",decode_base64("4oC+"),"\n";
for ([qq(a2)],[qq(c2a2)],[qw(203e)],[qw(3e 20)],[qw(1000)],[qw(00c7)],[qw(00ff 0000)]){
$s = pack "H*",@{$_};
printf "0x%s\t=> %s", join("",@{$_}), encode_base64($s);
}
perl6:
use Enc::MIME::Base64;
say encode_base64_str("\xa2");
say encode_base64_str("\x203e");
December 07, 2012 06:26
December 06, 2012
Perl 6 is built on lexical scopes. Variables, subroutines, constants and even types are looked up lexically first, and subroutines are only looked up in lexical scopes.
So it is only fitting that importing symbols from modules is also done into lexical scopes. I often write code such as
use v6;
# the main functionality of the script
sub deduplicate(Str $s) {
my %seen;
$s.comb.grep({!%seen{ .lc }++}).join;
}
# normal call
multi MAIN($phrase) {
say deduplicate($phrase)
}
# if you call the script with --test, it runs its unit tests
multi MAIN(Bool :$test!) {
# imports &plan, &is etc. only into the lexical scope
use Test;
plan 2;
is deduplicate('just some words'),
'just omewrd', 'basic deduplication';
is deduplicate('Abcabd'),
'Abcd', 'case insensitivity';
}
This script removes all but the first occurrence of each character given on the command line:
$ perl6 deduplicate 'Duplicate character removal'
Duplicate hrmov
But if you call it with the --test option, it runs its own unit tests:
$ perl6 deduplicate --test
1..2
ok 1 - basic deduplication
ok 2 - case insensitivity
Since the testing functions are only necessary in a part of the program — in a lexical scope, to be more precise –, the use statement is inside that scope, and limits the visibility of the imported symbols to this scope. So if you try to use the is function outside the routine in which Test is used, you get a compile-time error.
Why, you might ask? From the programmer's perspective, it reduces risk of (possibly unintended and unnoticed) name clashes the same way that lexical variables are safer than global variables.
From the point of view of language design, the combination of lexical importing, runtime-immutable lexical scopes and lexical-only lookup of subroutines allows resolving subroutine names at compile time, which again allows neat stuff like detecting calls to undeclared functions, compile-time type checking of arguments, and other nice optimizations.
But subroutines are only the tip of the iceberg. Perl 6 has a very flexible syntax, which you can modify with custom operators and macros. Those too can be exported, and imported into lexical scopes. Which means that language modifications are also lexically by default. So you can safely load any language-modifying extension, without running into danger that a library you use can't cope with it — the library doesn't even see the language modification.
So ultimately, lexical importing is another facet of encapsulation.
December 06, 2012 00:00
December 05, 2012
There’s much more to the developer experience of a language than its design, features and implementations. While the language and its implementations are perhaps the thing developers will spend most time with, the overall experience will also involve interaction with the community, reading documentation, using modules and employing various development tools. Thus, it’s important that Perl 6 make progress on these fronts too. Over the past year, we’ve taken some good steps forward in these areas; there’s now doc.perl6.org, the module ecosystem has grown, and the module installation tooling has improved. Another big step forward with regards to tooling – and the topic of this post – is that an interactive Perl 6 debugger is now available.
Running With The Debugger
The debugger has been included with the last few Rakudo * releases. If you have one of those, you’re all set. Just run perl6-debug instead of perl6. It takes the same set of options, so if your normal invocation involves, for example, using the -I flag to set the include path for modules, it’ll Just Work Like Usual. Of course, what happens next is entirely different. The debugger will show you each module it is loading, followed by placing you at the first interesting statement of the program, highlighted in yellow.
Note how it takes care to put you on the first line that actually does something, skipping the my statement above it (it’s getting increasingly smart about this).
The Basics
Hitting enter allows you to single-step through the program. At any point, you can look at variables, call methods on variables, or even evaluate expressions.
If you want to move statement by statement, but never descend into a function call or method call, type an s, followed by enter. To step out of the current sub (that is, run until it returns then break in its caller), use so to step out. To run the program until it hits an exception, just use r. Even at the point you get an exception, you can still access variables to try and dissect what went wrong.
One final variant, rt, will run until an exception is thrown, but handled. You’ll break at the point of the throw. This means you’re not disadvantaged in the debugger if you took care to handle exceptions well in your program; you can still break when they are thrown and use the debugger to help understand why. :-)
Breakpoints
Sometimes, you know exactly where the juicy stuff happens in your program that you wish to debug. If only you could just run until you got there. Turns out you can – that’s what breakpoints are for. We can add one, use r to run, and it will stop where we placed the breakpoint.
Note that you don’t have to type out the full name of the file you want to put the breakpoint in; any unambiguous substring of the name of a file that is loaded will be sufficient.
I won’t cover them here, but there are also tracepoints, which instead of breaking will log the value of an expression each time a certain place in the program is hit. Later, you can display the log. It’s like adding print statements, but without the print statement going in your code, removing the risk of them accidentally making it into a commit (‘cus we’ve all done that one, right? :-))
Regex and Grammar Debugging
When the debugger detects you are in a regex or grammar, it offers a little extra help. As well as allowing you to single-step your way through the regex, atom by atom, it also displays the match text, indicating what has been matched so far.
Here, you can see that the pattern already successfully matched SELECT, and is now looking for a literal * or will try to call the field list rule. If in a regex, which may backtrack, the match position jumps backwards when backtracking happens, so you can understand the backtracking behavior of the pattern.
Yes, Perl 5 Regexes Too!
Rakudo has some support for the :P5 adverb on regexes, which allows use of the Perl 5 regex syntax. Here the debugger is used in REPL mode (where you enter an expression, then can immediately debug it) to explore the difference between alternations in Perl 5 and Perl 6 (in Perl 5 they go left to right, in Perl 6 they have longest token matching semantics, such that it tries the thing that will match most characters first).
The debugger in REPL mode is great for exploring and understanding how things will execute (and as such can serve as a learning or teaching aid). Another use is for debugging modules without having to write a test script; just write a use statement in the debugger or you can even supply the module using the -M command line flag and the debugger will load it!
What About Funky Stuff, Like Macros?
That is, macros, BEGIN time, eval, and those other things where your Perl 6 program does the time warp again, doing a bit of runtime at compile time or compiling some more stuff at runtime. The debugger is built for it. If a macro is applied, the debugger will place you in it. Notice below how we’re still in the process of loading the second file, and did not get to the third yet – we really are debugging at BEGIN time!
Any lines of code that are stripped out by the macro are simply never hit at runtime. And what about statements in quasi blocks? The debugger will take you there, so you not only know what macro was applied, but can dig into exactly what it does too.
Just as bits of runtime happening at compile time work out fine, any code that gets eval‘d at runtime is also compiled with debug hooks, meaning that you can step straight into it and debug the evaluated code.
Written in Perl 6 and NQP!
You might think that writing a debugger must involve all kinds of low-level hackery. In fact, that’s not the case. The debug hooks mechanism is written in NQP, and the command line user interface is written in Perl 6. This is significant from a couple of angles. The first is the fact that we can write something like this without breaking the encapsulation of the compiler, but instead just by subclassing the Grammar, Actions and Compiler objects and twiddling with the AST. In fact, the debugger was built without any changes being required to Rakudo as it already existed. This provides important feedback on our compiler architecture – this time, very positive feedback. Things are extensible in the ways they were designed to be. The second is that writing so much of it in Perl 6 is a healthy bit of dogfooding – using the product in order to build further products. My hope is that, since most of what people would want to change is actually written in the Perl 6 part, it will feel quite hackable by the community at large.
And What Of Future Plans?
Various features are still to come: conditional breakpoints, dumping tracepoint output to a file, showing the path taken through a grammar to get to the current point, and various bits of configurability. The command line interface is nice, but of course having some extra options would be even nicer. I’m interested in a web-based interface, but also in integration with tools like Padre. There’s some work afoot on a common protocol for these things, which could make such integration possible without having to re-invent too many wheels. In the meantime, having an interactive debugger which is aware of and works well with a wide range of Perl 6 language features is a solid step forward. Happy debugging, and feature ideas (or patches ;-)) are welcome; here’s the GitHub repo!
December 05, 2012 00:01
December 04, 2012
Rakudo, the leading Perl6 implementation, is not perfect, and performance is a particularly sore subject. However, the pioneer does not ask ‘Is it fast?’, but rather ‘Is it fast enough?’, or perhaps even ‘How can I help to make it faster?’.
To convince you that Rakudo can indeed be fast enough, we’ll take a shot at a bunch of Project Euler problems. Many of those involve brute-force numerics, and that’s something Rakudo isn’t particularly good at right now. However, that’s not necessarily a show stopper: The less performant the language, the more ingenious the programmer needs to be, and that’s where the fun comes in.
All code has been tested with Rakudo 2012.11.
We’ll start with something simple:
Problem 2
By considering the terms in the Fibonacci sequence whose values do not exceed four million, find the sum of the even-valued terms.
The solution is beautifully straight-forward:
say [+] grep * %% 2, (1, 2, *+* ...^ * > 4_000_000);
Runtime: 0.4s
Note how using operators can lead to code that’s both compact and readable (opinions may vary, of course). We used
- whatever stars
* to create lambda functions
- the sequence operator (in its variant that excludes the right endpoint)
...^ to build up the Fibonacci sequence
- the divisible-by operator
%% to grep the even terms
- reduction by plus
[+] to sum them.
However, no one forces you to go crazy with operators – there’s nothing wrong with vanilla imperative code:
Problem 3
What is the largest prime factor of the number 600,851,475,143?
An imperative solution looks like this:
sub largest-prime-factor($n is copy) {
for 2, 3, *+2 ... * {
while $n %% $_ {
$n div= $_;
return $_ if $_ > $n;
}
}
}
say largest-prime-factor(600_851_475_143);
Runtime: 2.6s
Note the is copy trait, which is necessary as Perl6 binds arguments read-only by default, and that integer division div is used instead of numeric division /.
Nothing fancy going on here, so we’ll move along to
Problem 53
How many, not necessarily distinct, values of nCr, for 1 ≤ n ≤ 100, are greater than one-million?
We’ll use the feed operator ==> to factor the algorithm into separate steps:
[1], -> @p { [0, @p Z+ @p, 0] } ... * # generate Pascal's triangle
==> (*[0..100])() # get rows up to n = 100
==> map *.list # flatten rows into a single list
==> grep * > 1_000_000 # filter elements exceeding 1e6
==> elems() # count elements
==> say; # output result
Runtime: 5.2s
Note the use of the Z meta-operator to zip the lists 0, @p and @p, 0 with +.
The one-liner generating Pascal’s triangle has been stolen from Rosetta Code, another great resource for anyone interested in Perl6 snippets and exercises.
Let’s do something clever now:
Problem 9
There exists exactly one Pythagorean triplet for which a + b + c = 1000. Find the product abc.
Using brute force will work (solution courtesy of Polettix), but it won’t be fast (~11s on my machine). Therefore, we’ll use a bit of algebra to make the problem more managable:
Let (a, b, c) be a Pythagorean triplet
a < b < c
a² + b² = c²
For N = a + b + c it follows
b = N·(N - 2a) / 2·(N - a)
c = N·(N - 2a) / 2·(N - a) + a²/(N - a)
which automatically meets b < c.
The condition a < b gives the constraint
a < (1 - 1/√2)·N
We arrive at
sub triplets(\N) {
for 1..Int((1 - sqrt(0.5)) * N) -> \a {
my \u = N * (N - 2 * a);
my \v = 2 * (N - a);
# check if b = u/v is an integer
# if so, we've found a triplet
if u %% v {
my \b = u div v;
my \c = N - a - b;
take $(a, b, c);
}
}
}
say [*] .list for gather triplets(1000);
Runtime: 0.5s
Note the declaration of sigilless variables \N, \a, …, how $(…) is used to return the triplet as a single item and .list – a shorthand for $_.list – to restore listy-ness.
The sub &triplets acts as a generator and uses &take to yield the results. The corresponding &gather is used to delimit the (dynamic) scope of the generator, and it could as well be put into &triplets, which would end up returning a lazy list.
We can also rewrite the algorithm into dataflow-driven style using feed operators:
constant N = 1000;
1..Int((1 - sqrt(0.5)) * N)
==> map -> \a { [ a, N * (N - 2 * a), 2 * (N - a) ] }
==> grep -> [ \a, \u, \v ] { u %% v }
==> map -> [ \a, \u, \v ] {
my \b = u div v;
my \c = N - a - b;
a * b * c
}
==> say;
Runtime: 0.5s
Note how we use destructuring signature binding -> […] to unpack the arrays that get passed around.
There’s no practical benefit to use this particular style right now: In fact, it can easily hurt performance, and we’ll see an example for that later.
It is a great way to write down purely functional algorithms, though, which in principle would allow a sufficiently advanced optimizer to go wild (think of auto-vectorization and -threading). However, Rakudo has not yet reached that level of sophistication.
But what to do if we’re not smart enough to find a clever solution?
Problem 47
Find the first four consecutive integers to have four distinct prime factors. What is the first of these numbers?
This is a problem where I failed to come up with anything better than brute force:
constant $N = 4;
my $i = 0;
for 2..* {
$i = factors($_) == $N ?? $i + 1 !! 0;
if $i == $N {
say $_ - $N + 1;
last;
}
}
Here, &factors returns the number of prime factors. A naive implementations looks like this:
sub factors($n is copy) {
my $i = 0;
for 2, 3, *+2 ...^ * > $n {
if $n %% $_ {
++$i;
repeat while $n %% $_ {
$n div= $_
}
}
}
return $i;
}
Runtime: unknown (33s for N=3)
Note the use of repeat while … {…}, the new way to spell do {…} while(…);.
We can improve this by adding a bit of caching:
BEGIN my %cache = 1 => 0;
multi factors($n where %cache) { %cache{$n} }
multi factors($n) {
for 2, 3, *+2 ...^ * > sqrt($n) {
if $n %% $_ {
my $r = $n;
$r div= $_ while $r %% $_;
return %cache{$n} = 1 + factors($r);
}
}
return %cache{$n} = 1;
}
Runtime: unknown (3.5s for N=3)
Note the use of BEGIN to initialize the cache first, regardless of the placement of the statement within the source file, and multi to enable multiple dispatch for &factors. The where clause allows dynamic dispatch based on argument value.
Even with caching, we’re still unable to answer the original question in a reasonable amount of time. So what do we do now? We cheat and use Zavolaj – Rakudo’s version of NativeCall – to implement the factorization in C.
It turns out that’s still not good enough, so we refactor the remaining Perl code and add some native type annotations:
use NativeCall;
sub factors(int $n) returns int is native('./prob047-gerdr') { * }
my int $N = 4;
my int $n = 2;
my int $i = 0;
while $i != $N {
$i = factors($n) == $N ?? $i + 1 !! 0;
$n = $n + 1;
}
say $n - $N;
Runtime: 1m2s (0.8s for N=3)
For comparison, when implementing the algorithm completely in C, the runtime drops to under 0.1s, so Rakudo won’t win any speed contests just yet.
As an encore, three ways to do one thing:
Problem 29
How many distinct terms are in the sequence generated by ab for 2 ≤ a ≤ 100 and 2 ≤ b ≤ 100?
A beautiful but slow solution to the problem can be used to verify that the other solutions work correctly:
say +(2..100 X=> 2..100).classify({ .key ** .value });
Runtime: 11s
Note the use of X=> to construct the cartesian product with the pair constructor => to prevent flattening.
Because Rakudo supports big integer semantics, there’s no loss of precision when computing large numbers like 100100.
However, we do not actually care about the power’s value, but can use base and exponent to uniquely identify the power. We need to take care as bases can themselves be powers of already seen values:
constant A = 100;
constant B = 100;
my (%powers, %count);
# find bases which are powers of a preceeding root base
# store decomposition into base and exponent relative to root
for 2..Int(sqrt A) -> \a {
next if a ~~ %powers;
%powers{a, a**2, a**3 ...^ * > A} = a X=> 1..*;
}
# count duplicates
for %powers.values -> \p {
for 2..B -> \e {
# raise to power \e
# classify by root and relative exponent
++%count{p.key => p.value * e}
}
}
# add +%count as one of the duplicates needs to be kept
say (A - 1) * (B - 1) + %count - [+] %count.values;
Runtime: 0.9s
Note that the sequence operator ...^ infers geometric sequences if at least three elements are provided and that list assignment %powers{…} = … works with an infinite right-hand side.
Again, we can do the same thing in a dataflow-driven, purely-functional fashion:
sub cross(@a, @b) { @a X @b }
sub dups(@a) { @a - @a.uniq }
constant A = 100;
constant B = 100;
2..Int(sqrt A)
==> map -> \a { (a, a**2, a**3 ...^ * > A) Z=> (a X 1..*).tree }
==> reverse()
==> hash()
==> values()
==> cross(2..B)
==> map -> \n, [\r, \e] { (r) => e * n }
==> dups()
==> ((A - 1) * (B - 1) - *)()
==> say();
Runtime: 1.5s
Note how we use &tree to prevent flattening. We could have gone with X=> instead of X as before, but it would make destructuring via -> \n, [\r, \e] more complicated.
As expected, this solution doesn’t perform as well as the imperative one. I’ll leave it as an exercise to the reader to figure out how it works exactly ;)
That’s it
Feel free to add your own solutions to the Perl6 examples repository under euler/.
If you’re interested in bioinformatics, you should take a look at Rosalind as well, which also has its own (currently only sparsely populated) examples directory rosalind/.
Last but not least, some solutions for the Computer Language Benchmarks Game – also known as the Debian language shootout – can be found under shootout/.
You can contribute by sending pull requests, or better yet, join #perl6 on the Freenode IRC network and ask for a commit bit.
Have the appropriate amount of fun!
December 04, 2012 00:01
December 03, 2012
Introduction
This article aims to demonstrate how Whatever — one of the many interesting Perl 6 curiosities — could be useful to easily implement and use complex things like a layout manager. In a couple of words, a layout manager is the part of a graphical interface in charge of the spatial arrangement of objects like windows or widgets. For the sake of simplicity, the layout manager implemented in this article will comply with the following three rules:
- there are only two kinds of widgets: terminal or container, the latter can contain other widgets of either kind;
- a widget cannot be overlapped, except for containers which fully contain their sub-widgets; and
- only the height can be adjusted, this size can be either static, dynamic, or intentionally left unspecified.
Usage
From the user's point-of-view, this layout manager aims to be as easy to use as possible. For example, it shouldn't be hard to specify such typical interface below, inspired from a text-based program. In this example, the interface and body widgets are containers, all the others are terminals:
interface (X lines)
+----> +------------------------------------------+
| | menu bar (1 line) | body (remaining space)
| +------------------------------------------+ <----+
| | subpart 1 (1/3 of the remaining space) | |
| | | |
| | | |
| +------------------------------------------+ |
| | subpart 2 (remaining space) | |
| | | |
| | | |
| | | |
| | | |
| | | |
| +------------------------------------------+ <----+
| | status bar (1 line) |
+----> +------------------------------------------+
The user don't know what the remaining space is in advance because such an interface is arbitrary resizable. As a consequence it should be specified as a non-predefined value; this is where * — the Whatever object — comes in handy. This object is interesting for two reasons:
- from the user's point-of-view, the definition of non-static sizes is as simple as:
* / 3 for the subpart 1 (dynamic) and just * for the subpart 2 (unspecified); and
- from the developer's point-of-view, Perl 6 transforms automatically things like
$size = * / 3 into a closure: x → x / 3. Then, it could be called like a regular function: $size($x).
That way, the previous GUI can be transliterated into the following lines of code:
my $interface =
Widget.new(name => 'interface', size => $x, sub-widgets => (
Widget.new(name => 'menu bar', size => 1),
Widget.new(name => 'main part', size => *, sub-widgets => (
Widget.new(name => 'subpart 1', size => * / 3),
Widget.new(name => 'subpart 2', size => *))),
Widget.new(name => 'status bar', size => 1)));
Implementation
The drawing of terminal widgets is straightforward since most of the work is done by containers. Those are in charge to compute the remaining space as well as to uniformly distribute widgets that have an unspecified size:
class Widget {
has $.name;
has $.size is rw;
has Widget @.sub-widgets;
method compute-layout($remaining-space? is copy, $unspecified-size? is copy) {
$remaining-space //= $!size;
if @!sub-widgets == 0 { # Terminal
my $computed-size = do given $!size {
when Real { $_ };
when Callable { .($remaining-space) };
when Whatever { $unspecified-size };
}
self.draw($computed-size);
}
else { # Container
my @static-sizes = grep Real, @!sub-widgets».size;
my @dynamic-sizes = grep Callable, @!sub-widgets».size;
my $nb-unspecified = +grep Whatever, @!sub-widgets».size;
$remaining-space -= [+] @static-sizes;
$unspecified-size = ([-] $remaining-space, @dynamic-sizes».($remaining-space))
/ $nb-unspecified;
.compute-layout($remaining-space, $unspecified-size) for @!sub-widgets;
}
}
method draw(Real $size is copy) {
"+{'-' x 25}+".say;
"$!name ($size lines)".fmt("| %-23s |").say;
"|{' ' x 25}|".say while --$size > 0;
}
}
Here, any Callable object can be used to specify a dynamic size, as far as it takes the computed remaining space as argument. That means it is possible to specify more sophisticated dynamic size by passing a code Block. For example, { max(5, $^x / 3) } ensures the widget has a proportional size that can't decrease below 5.
Conclusion
It's time to check if this trivial layout manager works correctly both in Rakudo and Niecza, the two most advanced implementations of Perl 6. The following test is rather simple, it creates and draws an interface, then resize it and draws it again:
my $interface =
Widget.new(name => 'interface', size => 11, sub-widgets => (
Widget.new(name => 'menu bar', size => 1),
Widget.new(name => 'main part', size => *, sub-widgets => (
Widget.new(name => 'subpart 1', size => * / 3),
Widget.new(name => 'subpart 2', size => *))),
Widget.new(name => 'status bar', size => 1)));
$interface.compute-layout; # Draw
$interface.size += 3; # Resize
$interface.compute-layout; # Redraw
The results before and after resizing are respectively displayed below. They are close enough from the initial mockup, n'est-ce pas?
+-------------------------+ +-------------------------+
| menu bar (1 lines) | | menu bar (1 lines) |
+-------------------------+ +-------------------------+
| subpart 1 (3 lines) | | subpart 1 (4 lines) |
| | | |
| | | |
+-------------------------+ | |
| subpart 2 (6 lines) | +-------------------------+
| | | subpart 2 (8 lines) |
| | | |
| | | |
| | | |
| | | |
+-------------------------+ | |
| status bar (1 lines) | | |
| |
+-------------------------+
| status bar (1 lines) |
Finally, the implementation of such a flexible program is really simple in Perl 6: everything is already there, in the core language. Obviously, this trivial layout manager isn't ready for prime-time since a lot of things are missing: sanity checks, multiple dimensions, … but those are left as exercises to you, the reader ;) For any questions or comments, feel free to meet Perl 6 fellows on IRC (#perl6 on freenode).
Bonus
As seen previously, $!size can be Whatever, but it can't be whatever you want. For example, a negative Real or a string are not correct values. Once again Perl 6 provides a simple yet powerful feature: constrained types. In a couple of words this permits to define new types from a set of constraints:
subset PosReal of Real where * >= 0;
subset Size where { .does(PosReal)
or .does(Callable) and .signature ~~ :(PosReal --> PosReal)
or .does(Whatever) };
has Size $.size is rw;
December 03, 2012 00:01
December 02, 2012
Perl 6 has great support for functions. It packs function signatures full with awesome, and lets you have your cake and eat it a couple of times over with all the ways you can specify a function. You can specify parameter types, optional parameters, named parameters, and even those cool where clauses. If I didn’t know better, I’d suspect Perl 6 was compensating for some predecessor’s rather rudimentary handling of parameters. (*cough* @_ *cough*)
Among all these other things, Perl 6 also allows you to define functions without naming them.
sub { say "lol, I'm so anonymous!" }
How is this useful? If you can’t name the function, you can’t call it, right? Wrong.
You can store the function in a variable. Or return it from another function. Or pass it to another function. In fact, when you don’t name your function, the focus becomes much more what code you’re going to run later. Like an executable “to do”-list.
Of course, Perl 5 has anonymous functions, too. With exactly the same syntax, even. In fact, all the big languages do anonymous functions, according to this list of languages on Wikipedia. Except, it seems, the historically significant languages C and Pascal. And the more modern but lumbering Java. “Planned for Java 8″. Haha, Java, catch up! Even C++ has them now.
How important are anonymous functions? Very. In the 1930s, Alan Turing showed how all computer processes could be simulated using just a pre-programmed machine that looks like a tape recorder, reading and writing values on a really long tape. (The Turing Machine.) Meanwhile, across the Atlantic, Alonzo Church showed how all computer processes could be simulated using just anonymous functions, no tape recorder required. (Lambda calculus.) It’s all quite elegant.
Later languages like Lisp and Scheme lean heavily on anonymous functions as a key component in the language. And lately a scrappy language called JavaScript, which also leans heavily on anonymous functions, has taken over the world while we were all busy surfing the web.
But let’s talk possibilities here. What can anonymous functions do for us? And how would it look in Perl 6?
Well, take sorting as a famous example. You could imagine Perl 6 having a sort_lexicographically function and a sort_numerically function. But it doesn’t. It has a sort function. When you want it to sort in a certain way, you just pass an anonymous function to it.
my @sorted_words = @words.sort({ ~$_ });
my @sorted_numbers = @numbers.sort({ +$_ });
(Technically, those are blocks, not functions. But the difference isn’t significant if you’re not planning to return anywhere inside.)
And of course it goes further than just those two sort orders. You could sort by shoe size, or maximum ground speed, or decreasing likelihood of spontaneous combustion. All because you can pass in any logic as an argument. Object-oriented people are very proud of this pattern, and call it “dependency injection”.
Come to think of it, map and grep and reduce all depend on this kind of function-passing. We sometimes refer to passing functions to functions as “higher order programming”, as if it was only something people with special privileges should be doing. But in fact it’s a very useful and broadly applicable technique.
The above examples all run the anonymous functions as part of their own execution. But there’s no need to restrict ourselves to this. We can create functions, return them, and then run them later:
sub make_surprise_for($name) {
return sub { say "Sur-priiise, $name!" };
}
my $reveal_surprise = make_surprise_for("Finn"); # nothing happens, yet
# ...wait for it...
# ...wait...
# ...waaaaaaait...
$reveal_surprise(); # "Sur-priiise, Finn!"
The function in $reveal_surprise remembers the value of $name even though the original function passing it in has exited long ago. That’s pretty nice. This effect is referred to as the anonymous function closing over the variable $name. But there’s no need to get technical — the long and short of it is “it’s awesome”.
And in fact, it feels quite natural if we just look at anonymous functions alongside other staple storage mechanisms such as arrays and hashes. All of these can be stored in variables, passed as arguments or returned from functions. An anonymous array allows you to store a sequence of things for later. An anonymous hash allows you to store mappings/translations of things for later. An anonymous function allows you to store calculations or behavior for later.
Later this month, I’ll go through how to exploit dynamic scoping in Perl 6 to create nice DSL-y interfaces. We’ll see how anonymous functions come into play there as well.
December 02, 2012 00:01
