<!--- Title -->
Description
-----------
<!--- Describe your changes in detail. -->
This PR enables the coreMQTT library to resend unacked publishes on an
unclean session connection.
Following is a brief summary of changes:
1. Add a new API `MQTT_InitRetransmits` that will initialise the context
to handle publish retransmits on an unclean session connection
2. Add signatures of callback function pointers that users will define
in order to:
a. copy and store outgoing publishes
b. retrieve copied publish on an unclean session connection to resend
c. clear a copied publish when a `PUBACK`/`PUBREC` is received
d. clear all copied publishes on a clean session connection
3. Update the API's to check if callback's are defined and implement
resend publishes as required.
Following are the specifics of the changes:
1. Add 3 new MQTTStatus_t values: MQTTPublishStoreFailed,
MQTTPublishRetrieveFailed and MQTTPublishClearAllFailed
2. Update `MQTTContext_t` to hold the callback function pointers
a. `MQTTRetransmitStorePacket storeFunction`
b. `MQTTRetransmitRetrievePacket retrieveFunction`
c. `MQTTRetransmitClearPacket clearFunction`
d. `MQTTRetransmitClearAllPackets clearAllFunction`
3. Update the `MQTT_Status_strerror` function to handle the new
`MQTTStatus_t` values
4. Add a new API function `MQTT_InitRetransmits` that will initialise
the new callback functions in the `MQTTContext_t`
5. Add this API to the core_mqtt.h file to make it available to users
6. Modify `MQTT_Publish`
a. copy the outgoing publish packet in form of an array of
`TransportOutVector_t` if the callback if defined
b. if copy fails then bubble up corresponding error status code
7. Modify `MQTT_ReceiveLoop`
a. on receiving a `PUBACK`/`PUBREC` clear the copy of that particular
publish after the state of the publish record has been successfully
updated, if the callback if defined
8. Modify `MQTT_Connect`
a. on a clean session clear all the copies of publishes stored if the
callback is defined
b. if clear all fails then bubble up corresponding error status code
c. on an unclean session get the packetID of the unacked publishes and
retrieve the copies of those if the callback is defined
d. if retrieve fails then bubble up corresponding error status code
Approaches Taken
---------------
- To let user know about the changes we have made we will add them to a
changelog and have a minor version bump
- To be in line with the zero copy principle in our library we chose to
provide and retrieve the publish packets for storing and resending in
form of an array of `TransportOutVector_t`
- Code is written in a way that on receiving a `PUBACK`/`PUBREC` the
copy will be cleared after the state of the publish record is changed so
that if state update fails the copy won't be cleared. Otherwise if the
state does not change and the copy is cleared then when a connection is
made with an unclean session there will be a retrieve fail as the system
is in an inconsistent state.
- We are storing the copies of the publishes with the Duplicate flag set
this is because on retrieving the packet we will get it in the form of a
`TransportOutVector_t` that holds the data in a `const` pointer which
cannot be changed after retrieving.
Pending Tasks
---------------
- [ ] Changelog
- [ ] Minor version bump
- [x] Doxygen example for the new API
- [x] Better API Names
- [x] Unit Test Updates
- [x] CBMC Proof
---------
Co-authored-by: Dakshit Babbar <dakshba@amazon.com>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: AniruddhaKanhere <60444055+AniruddhaKanhere@users.noreply.github.com>
<!--- Title -->
Description
-----------
<!--- Describe your changes in detail. -->
Following is a brief summary of changes:
1. Check the connected flag before doing any send operation on the
connection
2. Make all the APIs that do send operations, thread safe
3. Update the connected flag within MQTT_Disconnect regardless of the
return status of the send operation
Following are the specifics of the changes:
1. Add 3 new MQTTStatus_t values: MQTTStatusConnected,
MQTTStatusNotConnected and MQTTStatusDisconnectPending
2. Added 1 new MQTTConnectionStatus_t value: MQTTDisconnectPending
3. Update the MQTT_Status_strerror function to handle the new
MQTTStatus_t values
4. Add a new API function MQTT_CheckConnectStatus() that will check the
value of the context→connectStatus flag safely.
5. Add this API to the core_mqtt.h file to make it available to users
6. Check the connected flag before doing any Send operation (following
API's are updated)
a. sendPublishAcks
b. MQTT_Connect
c. MQTT_Subscribe
d. MQTT_Publish
e. MQTT_Ping
f. MQTT_Unsubscribe
g. MQTT_Disconnect
7. Use the MQTT_PRE_STATE_UPDATE_HOOK() and
MQTT_POST_STATE_UPDATE_HOOK() to make the send APIs thread safe
8. The connect status is set to MQTTDisconnectPending whenever a
transport send or receive function returns a negative error code
9. `const` keyword for the the MQTTStatus_t is removed in the input
parameters for the receive functions as we need to update the connection
status when the receive function returns a negative error code
Relevant Explanations
---------------
- MQTT_PRE_SEND_HOOK(): The Pre and Post Send hook Macros are not
required now, as the sending logic will be within the pre and post state
update hook itself. (because we cannot allow other threads to change the
connection state of the application until a send operation is complete).
- I have split the handleSessionResumption function. The part of that
function which was handling the clean session has been added within the
mutex calls in the [MQTT_Connect
API](https://github.com/FreeRTOS/coreMQTT/pull/305/files#diff-2534a3c0229ae9af3801f2a5c6a24eeef2cd0a686671f0371a11d2718ba4fdd6R2828)
and the unclean session part is handled by this new function that is
[called outside the mutex
calls](https://github.com/FreeRTOS/coreMQTT/pull/305/files#diff-2534a3c0229ae9af3801f2a5c6a24eeef2cd0a686671f0371a11d2718ba4fdd6R2866).
Pending Tasks
---------------
- [ ] Doxygen example for the new API
- [x] Unit Test Updates
- [x] CBMC Proof
---------
Co-authored-by: Dakshit Babbar <dakshba@amazon.com>
Co-authored-by: GitHub Action <action@github.com>
Description
-----------
The `mqtt.tag` was generated at `coreMQTT/docs/doxygen/output/mqtt.tag`.
The
[doxygen-generation](https://github.com/FreeRTOS/CI-CD-Github-Actions/tree/main/doxygen-generation)
GH action deploys the content of `coreMQTT/docs/doxygen/output/html` and
as a result, the `mqtt.tag` was not getting deployed.
This change updates the location of `mqtt.tag` to
`coreMQTT/docs/doxygen/output/html/mqtt.tag` to ensure that it gets
deployed by the doxygen-generation GH action.
Test Steps
-----------
Generated the doxygen documentation locally.
Checklist:
----------
<!--- Go over all the following points, and put an `x` in all the boxes
that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
- [x] I have tested my changes. No regression in existing tests.
- [NA] I have modified and/or added unit-tests to cover the code changes
in this Pull Request.
Related Issue
-----------
NA.
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
---------
Signed-off-by: Gaurav Aggarwal <aggarg@amazon.com>
* Update the CHANGELOG.md to include v2.3.1 information
* Update version number to "v2.3.1+" in main branch in public header file macro, manifest.yml and config.doxyfile.
<!--- Title -->
Description
-----------
* Update the release action for version number include the following
files
- docs/doxygen/config.doxyfile - PROJECT_NUMBER
- manifest.yml file - version
- source file - version header
- core_mqtt.h - version number
* Add version number check in "Create ZIP and verify package for release
asset" steps. Including the following
- docs/doxygen/config.doxyfile - PROJECT_NUMBER
- manifest.yml file - version
- source file - version header
- core_mqtt.h - version number
* Update all the version number to "v2.3.0+" and "\<DEVELOPMENT
BRANCH\>"
Test Steps
-----------
Using release action to create release should update the following
* source/include/core_mqtt.h version number
* source files header version number
* doxygen version number
* manifest.yml number
* SBOM file
Tested in personal fork without problem :
https://github.com/FreshDevGo/coreMQTT/actions/runs/9885707328/job/27304218049
Test with wrong source file version number :
https://github.com/FreshDevGo/coreMQTT/actions/runs/9885727002/job/27304274003
Test with wrong manifest.yml version number :
https://github.com/FreshDevGo/coreMQTT/actions/runs/9885726029/job/27304270303
Test with wrong doxygen version number :
https://github.com/FreshDevGo/coreMQTT/actions/runs/9885723302/job/27304269170
Test with wrong version number macro in core_mqtt.h :
https://github.com/FreshDevGo/coreMQTT/actions/runs/9885724835/job/27304268841
Checklist:
----------
<!--- Go over all the following points, and put an `x` in all the boxes
that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
- [x] I have tested my changes. No regression in existing tests.
- [ ] ~~I have modified and/or added unit-tests to cover the code
changes in this Pull Request.~~
Related Issue
-----------
<!-- If any, please provide issue ID. -->
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
<!--- Title -->
Description
-----------
This PR update changelog, version numbers and .md (doxygen, size table)
files for release
Test Steps
-----------
<!-- Describe the steps to reproduce. -->
Checklist:
----------
<!--- Go over all the following points, and put an `x` in all the boxes
that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
- [ ] I have tested my changes. No regression in existing tests.
- [ ] I have modified and/or added unit-tests to cover the code changes
in this Pull Request.
Related Issue
-----------
<!-- If any, please provide issue ID. -->
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
* Initialise variables in test before use
* Init variables in serializer tests
* Update uncrustify runner OS version
* Update doxygen config and doxygen used to latest
* Update brief and remove unused return description
* Update python version for latest ubuntu image
* Add SPDX identifier to source files
* Add license information to CBMC stub
* Fix spellcheck and the formatting
* Update version numbers and .md files for reelase
Co-authored-by: jasonpcarroll <23126711+jasonpcarroll@users.noreply.github.com>
* Update the timout check in the send function
* Fix direction of check
* Allow processing of data in the buffer
* Fix formatting and unit-tests
* Update comment to clarify control flow
* Fix Disconnect CBMC proof and update size table
* Fix formatting and CBMC proofs
* Fix last CBMC proof
* Fix broken unit tests and add branch coverage
* Update ci.yml (#197)
* Update ci.yml
* Add main branch in the CI YAML
* Add cancel callback API (#196)
* Add cancel callback command
* Update the function name
* Fix formatting
* Update memory estimates
* Add State to MQTT_ProcessLoop so that it can be called in a non-blocking manner (#198)
* Add stateful process-loop function
* Add extra checks; fix bugs and add description of functions
* Add index based stateful processloop
* Clean up
* Renamed functions to make them more coherent with their function
* Remove unused function declarations
* Fixed failing CI checks from previous commits except unit-test
* Fixed spell check and updated size-table
* Fix CBMC proofs
* Empty-Commit to trigger CBMC proofs
* Fix loop unwinding values in the Makefile
* Add upper bound on the buffer size of MQTT
* Increase minimum limit on buffer size to >0
* Add upper bound on the size of the buffer as well
* CBMC: Add memmove stub to accelerate coverage
The commit adds a stub for memmove accelerate CBMC coverage
calculation. Without this stub, coverage for `MQTT_ProcessLoop` and
`MQTT_ReceiveLoop` fails to converge (gets stuck generating the SAT
formula for the memmove in `receiveSingleIteration`). This stub
checks that src and dst are nonnull pointers and havocs dst.
* Fix formatting
Co-authored-by: Aniruddha Kanhere <ubuntu@ip-172-31-25-12.us-west-2.compute.internal>
Co-authored-by: Mark R. Tuttle <mrtuttle@amazon.com>
* Remove the use of common buffer (#199)
* Make publish use internal buffers
* Fix comment about packet ID
* Try a different approach for ping and disconnect
* Use writev and flush in conjuction with send
* Update the publish method to use vectors
* Add vectored IO to all functions
* Fix formatting
* Reduce complexity score
* Fix spell check and complexity score
* Fix breaking build
* Add doxygen comments
* Fix doxygen part 2
* Doxygen fix part 3
* Fix doxygen part 4
* Fix some checks
* Fix memory tables
* Fix some small errors
* Fix compiler warnings and breaking CI checks from previous commit
* Fix spell check and doxygen
* Fix a couple of CBMC proofs
* Fix ping and publish proofs
* Update the function name
* Fix more CBMC proofs
* Fix MQTT Connect proof
* Add unwinding loops
* Fix last CBMC proof
* Fix formatting
* Update the Subscribe and Unsubscribe functions
* Fix formatting and doxygen checks
* Fix broken CBMC proofs
* Fix memory statistic table
* Revert changes from serializer source
* update comments to clarify write requirements
* Add a note for write function pointer
* Fix spell check
* Update changelog (#202)
* Update MQTT logging so that log levels of the library do not leak (#205)
* Replace publish state arrays with pointers
Added an MQTT_InIt function for QoS > 0 publishes
Fixed functions which were dealing with state arrays
* Fix CI checks and clean up
* Fix CBMC proofs
* Fix sub and unsub CBMC proofs
* Fix remaining proofs
* Fix remaining CI checks
* Fix spell check
* Minor typo fix (#209)
* Update core_mqtt.h (#208)
* Update core_mqtt.h
* Update core_mqtt.h
Co-authored-by: Aniruddha Kanhere <60444055+AniruddhaKanhere@users.noreply.github.com>
* Update documentation and Doxygen comments in the source (#206)
* Update comments of the MQTT_InIt function
* Updating documentation of more of functions
* Fix formatting and spell check
* Update core_mqtt.h
* Add hooks to the source code (#200)
* Add mutex hooks
* Clean up of code
* Add doxygen comments and fix spell check
* Fix LogError call
* Fix formatting and memory table
* Fix dereference failure
* Update the hook names
* Fix broken builds
* Update the macros and variables
* Reword the briefs of hooks and uncrustify
* Fir formatting
* Protect get packet ID
* Fix formatting
* Fix Unit tests (#212)
* Fix Unit tests
* Update unit tests after new changes
* Fix more UT
* add dummy calls to the transport
* fix build error
* Remove usused variables
* Remove unsused variables
* Remove usused variables
* Unsued parameter
* Fix ut failure
* Fix uninitialized unit test variables
* Fix ut expectation
* Fix unit-tests
* Fix unit test uninitialized variable
* increase unit test coverage
* increase unit test coverage
* Fix unit test build
* State coverage 100%
* Serializer 100% coverage
* 100% UT coverage
* Fix formatting
* Fix size table
* Address PR comments
Co-authored-by: Aniruddha Kanhere <60444055+AniruddhaKanhere@users.noreply.github.com>
* Remove unnecessary ternaries (#211)
Co-authored-by: Aniruddha Kanhere <60444055+AniruddhaKanhere@users.noreply.github.com>
* Fix misra deviations (#213)
* MISRA compliance update 1
* Fix MISRA errors
* Zero MISRA violations; 13 suppressed
* Fix CI checks
* Update MISRA.md
* Remove deviations
* Fix MISRA.md file
* Fix bug - wasn't incrementing vector length properly.
* Remove unwanted files
* Update comment to clarify the control flow of UT
* Remove 'dev' branch from the CI checks
* Update horrid threshold = 10
Co-authored-by: Aniruddha Kanhere <ubuntu@ip-172-31-25-12.us-west-2.compute.internal>
Co-authored-by: Mark R. Tuttle <mrtuttle@amazon.com>
Co-authored-by: jasonpcarroll <23126711+jasonpcarroll@users.noreply.github.com>
Co-authored-by: alfred gedeon <28123637+alfred2g@users.noreply.github.com>
Co-authored-by: Archit Gupta <71798289+archigup@users.noreply.github.com>
* generate PINREQ packets on idle input or output.
* changes addressing Paul's feedback.
* changes to reflect feedback from Paul and Cobus.
* further changes after discussions with Paul.
* address issues raised by static analysis and formatting.
* update documentation and unit tests.
* use if else to clarify.
* remove stale variable.
* fix logical error.
* increment MQTT_TIMER_CALLS_PER_ITERATION by 1.
* add lastpackettxtime to the lexicon.
* use a different uncrustify config and add rx to the lexicon.
* update unit test to acheive coverage and correct memory size expectations and default tieout settings.
* cover case where keep alive interval is greater than the tx timeout.
* use correct units.
* dont pre-set waitingForPingResp to true, we want handleKeepAlive to trigger that.
* expect success on the new subtest.
* add additional test cases to cover two new branches.
* remove unused variable.
* MISRA compliance change.
* try to make both MISRA and uncrustify happy.
* try different version of uncrustify.
* Set PACKET_RX_TIMEOUT_MS to 30000U to match comment
Co-authored-by: Paul Bartell <pbartell@amazon.com>
* Update version numbers
* Update CHANGELOG.md
* Add guards for C++ linkage
* Link to Memory estimates markdown from README
* Make possible to override CMake C Standard for tests
Update the sub-section of the Design page of the library API reference to provide detailed information about the role of runtime timeout and compile-time timeout configurations in the behavior of MQTT_Connect, MQTT_ProcessLoop and MQTT_ReceiveLoop functions.
* Fix issue of data recv being interrupted
* Rename elapsedTimeMs variable to its express its new meaning
* Use configuration const for recvExact timeout
* Remove timeout check from discardPacket and address CI check failures
* Fix more CI check failures
* Remove another unused local variable
* Re-instate timeout in discard to reduce scope of changes
* Remove unused variable again
* Fix failing unit test
* Rename new config macro, and attempt to fix CBMC failures
* Doc: Improvement suggestions from code review
Co-authored-by: Muneeb Ahmed <54290492+muneebahmed10@users.noreply.github.com>
* Fix quality check failures
* Add test case to check partial network reads with zero timeout duration for ProcessLoop
* style: Improving naming
* Address complexity failure
* Address comments
* Doc: Add blocking time equation of Receive/ProcessLoop functions in their API doc
* Improvement in API doc
* Set MQTT_RECV_POLLING_TIMEOUT_MS so that recvExact runs in one iteration always for cbmc.
* doc: Add information about zero return value for Transport_Recv_t
* fix: prevent possibility of infinite loop in timeout logic of ProcessLoop
* style: Minor changes
* hygiene: minor name fix
* fix: Possibility of infinite loop in sendPacket
* Add the new configuration to doxygen
* test: Add mock transport send function that always returns zero
* fix: Issues in sendPacket and sendPublish
* test: add test for sendPacket timeout
* Update Timer Overflow test
* test: temporarily comment out unused variable
* test: fix the timer overflow test
* Address review comments
* style: make log messages concise
Co-authored-by: Muneeb Ahmed <54290492+muneebahmed10@users.noreply.github.com>
Co-authored-by: Muneeb Ahmed <54290492+muneebahmed10@users.noreply.github.com>
Co-authored-by: Sarena Meas <sarem@amazon.com>
### Problem
The `MQTT_ProcessLoop` and `MQTT_ReceiveLoop` read incoming MQTT packet payload over the network by calling the `recvExact` function. The `recvExact` function can be called multiple times to read the expected number of bytes for the MQTT packet but it also implements a timeout functionality of receiving the expected number of payload within the timeout value passed to the function.
This causes problems when the `Transport_Recv` call returns less than requested number of bytes, and there is a timeout (for example, when calling `MQTT_ProcessLoop` with 0ms duration) which causes the function to assume failure instead of reading the remaining payload of the MQTT packet by calling `Transport_Recv` again. Thus, in such cases, the MQTT connection is severed prematurely even though there is a high probability of receiving the remaining bytes of the MQTT packet over the network.
### Solution
Instead of implementing a timeout on the entire duration of receiving the expected number of remaining MQTT packet bytes in `recvExact`, the use of timeout is being changed to be relevant only on the total time of receiving 0 bytes over the network over multiple calls to `Transport_Recv`.
As this modified meaning of the timeout duration is now unrelated to the timeout duration that the `MQTT_ProcessLoop` or `MQTT_ReceiveLoop` functions are called, a new configuration constant for the `recvExact` timeout value, `MQTT_RECV_POLLING_TIMEOUT_MS`, has been added to the library which will carry a default value of 10ms.
Co-authored-by: Sarena Meas <sarem@amazon.com>
This follows changes from FreeRTOS/FreeRTOS@398abba. The const qualifier is removed from send/recv because there are transport implementations that require a member of the network context to be modified such as in the case of mbedtls.
* Update code sizes after adding verison number
* Round as much as possible when multiplying by 1024
* Add code size when no optimisation is used
* Update totals
* Change MQTT RC1 to coreMQTT
* Add link to gcc arm toolchain
* Update link
* Use english spellings
Co-authored-by: Gary Wicker <14828980+gkwicker@users.noreply.github.com>
